Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Dan Williams]

On Fri, Jul 3, 2015 at 11:30 AM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> On Sat, Jun 27, 2015 at 04:45:25PM -0700, Dan Williams wrote:
>> On Mon, Mar 30, 2015 at 4:20 PM, Dmitry Torokhov
>> <dmitry.torokhov@gmail.com> wrote:
>> > Some devices take a long time when initializing, and not all drivers are
>> > suited to initialize their devices when they are open. For example,
>> > input drivers need to interrogate their devices in order to publish
>> > device's capabilities before userspace will open them. When such drivers
>> > are compiled into kernel they may stall entire kernel initialization.
>> >
>> > This change allows drivers request for their probe functions to be
>> > called asynchronously during driver and device registration (manual
>> > binding is still synchronous). Because async_schedule is used to perform
>> > asynchronous calls module loading will still wait for the probing to
>> > complete.
>> >
>> > Note that the end goal is to make the probing asynchronous by default,
>> > so annotating drivers with PROBE_PREFER_ASYNCHRONOUS is a temporary
>> > measure that allows us to speed up boot process while we validating and
>> > fixing the rest of the drivers and preparing userspace.
>> >
>> > This change is based on earlier patch by "Luis R. Rodriguez"
>> > <mcgrof@suse.com>
>> >
>> > Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
>> > ---
>> >  drivers/base/base.h    |   1 +
>> >  drivers/base/bus.c     |  31 +++++++---
>> >  drivers/base/dd.c      | 149 ++++++++++++++++++++++++++++++++++++++++++-------
>> >  include/linux/device.h |  28 ++++++++++
>> >  4 files changed, 182 insertions(+), 27 deletions(-)
>>
>> Just noticed this patch.  It caught my eye because I had a hard time
>> getting an open coded implementation of asynchronous probing to work
>> in the new libnvdimm subsystem.  Especially the messy races of tearing
>> things down while probing is still in flight.  I ended up implementing
>> asynchronous device registration which eliminated a lot of complexity
>> and of course the bugs.  In general I tend to think that async
>> registration is less risky than async probe since it keeps wider
>> portions of the traditional device model synchronous
>
> but its not see -DEFER_PROBE even before async probe.

Except in that case you know probe has been seen by the driver at
least once.  So I see that as less of a surprise, but point taken.

>> and leverages the
>> fact that the device model is already well prepared for asynchronous
>> arrival of devices due to hotplug.
>
> I think this sounds reasonable, do you have your code upstream or posted?

Yes, see nd_device_register() in drivers/nvdimm/bus.c

> If not will you be at Plumbers?

Yes.

> Maybe we shoudl talk about this as although
> ChromeOS already likely already jumped on async probe we should address a
> way forward and path forward for other distributions and I don't think anyone
> is looking too much into it. async probe came to Linux for two reasons:
>
>  * chromeos wanting it
>  * an incorrect systemd assumption on how the driver core works
>
> So long term we still need to address the systemd approach, are they going
> to be defaulting now to async probe for all modules? How about for built-ins?
>
> We should talk about this and maybe at plumbers.
>
>> Splitting the "initial probe" from
>> the "manual probe" case seems like a recipe for confusion.
>
> If you can come up with pros / cons on both strategies it'd be
> valuable.

The problem I ran into was needing to remove devices that still had
yet to be probed and not being able to use registration completion vs
the device_lock() to effectively synchronize the sub-system.

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Luis R. Rodriguez]

On Sat, Jul 04, 2015 at 07:09:19AM -0700, Dan Williams wrote:
> On Fri, Jul 3, 2015 at 11:30 AM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> > On Sat, Jun 27, 2015 at 04:45:25PM -0700, Dan Williams wrote:
> >> On Mon, Mar 30, 2015 at 4:20 PM, Dmitry Torokhov
> >> <dmitry.torokhov@gmail.com> wrote:
> >> > Some devices take a long time when initializing, and not all drivers are
> >> > suited to initialize their devices when they are open. For example,
> >> > input drivers need to interrogate their devices in order to publish
> >> > device's capabilities before userspace will open them. When such drivers
> >> > are compiled into kernel they may stall entire kernel initialization.
> >> >
> >> > This change allows drivers request for their probe functions to be
> >> > called asynchronously during driver and device registration (manual
> >> > binding is still synchronous). Because async_schedule is used to perform
> >> > asynchronous calls module loading will still wait for the probing to
> >> > complete.
> >> >
> >> > Note that the end goal is to make the probing asynchronous by default,
> >> > so annotating drivers with PROBE_PREFER_ASYNCHRONOUS is a temporary
> >> > measure that allows us to speed up boot process while we validating and
> >> > fixing the rest of the drivers and preparing userspace.
> >> >
> >> > This change is based on earlier patch by "Luis R. Rodriguez"
> >> > <mcgrof@suse.com>
> >> >
> >> > Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
> >> > ---
> >> >  drivers/base/base.h    |   1 +
> >> >  drivers/base/bus.c     |  31 +++++++---
> >> >  drivers/base/dd.c      | 149 ++++++++++++++++++++++++++++++++++++++++++-------
> >> >  include/linux/device.h |  28 ++++++++++
> >> >  4 files changed, 182 insertions(+), 27 deletions(-)
> >>
> >> Just noticed this patch.  It caught my eye because I had a hard time
> >> getting an open coded implementation of asynchronous probing to work
> >> in the new libnvdimm subsystem.  Especially the messy races of tearing
> >> things down while probing is still in flight.  I ended up implementing
> >> asynchronous device registration which eliminated a lot of complexity
> >> and of course the bugs.  In general I tend to think that async
> >> registration is less risky than async probe since it keeps wider
> >> portions of the traditional device model synchronous
> >
> > but its not see -DEFER_PROBE even before async probe.
> 
> Except in that case you know probe has been seen by the driver at
> least once.  So I see that as less of a surprise, but point taken.
> 
> >> and leverages the
> >> fact that the device model is already well prepared for asynchronous
> >> arrival of devices due to hotplug.
> >
> > I think this sounds reasonable, do you have your code upstream or posted?
> 
> Yes, see nd_device_register() in drivers/nvdimm/bus.c

It should be I think rather easy for Dmitry to see if he can convert this input
driver (not yet upstream) to this API and see if the same issues are fixed.
This however does not address systemd's assumption over detachment of module
load and probe. The inherent problem there was the timeout implemented and
carried in systemd over the worker that uses modlib to load modules. Upon
review the code was complex enough already and surely increasing the timeout
helps but that doesn't address all issues with a general timeout in place.
At SUSE we ended up not using a timeout for kmod built-in commands. That
leaves the original timeout purpose in place. The code for async probe was
not put in the kernel though but since its now upstream we should be able
to replace that userspace systemd work around with async probe, but systemd
folks would need to decide what they want to do. For full gory details of
this refer to:

https://bugzilla.novell.com/show_bug.cgi?id=889297

> > If not will you be at Plumbers?
> 
> Yes.

Great, Tom, Dmitry, will you be at Plumbers?

> > Maybe we shoudl talk about this as although
> > ChromeOS already likely already jumped on async probe we should address a
> > way forward and path forward for other distributions and I don't think anyone
> > is looking too much into it. async probe came to Linux for two reasons:
> >
> >  * chromeos wanting it
> >  * an incorrect systemd assumption on how the driver core works
> >
> > So long term we still need to address the systemd approach, are they going
> > to be defaulting now to async probe for all modules? How about for built-ins?
> >
> > We should talk about this and maybe at plumbers.
> >
> >> Splitting the "initial probe" from
> >> the "manual probe" case seems like a recipe for confusion.
> >
> > If you can come up with pros / cons on both strategies it'd be
> > valuable.
> 
> The problem I ran into was needing to remove devices that still had
> yet to be probed and not being able to use registration completion vs
> the device_lock() to effectively synchronize the sub-system.

Interesting, what cases would this happen under?

  Luis

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Dmitry Torokhov]

On Tue, Jul 07, 2015 at 01:23:15AM +0200, Luis R. Rodriguez wrote:
> On Sat, Jul 04, 2015 at 07:09:19AM -0700, Dan Williams wrote:
> > On Fri, Jul 3, 2015 at 11:30 AM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> > > On Sat, Jun 27, 2015 at 04:45:25PM -0700, Dan Williams wrote:
> > >> On Mon, Mar 30, 2015 at 4:20 PM, Dmitry Torokhov
> > >> <dmitry.torokhov@gmail.com> wrote:
> > >> > Some devices take a long time when initializing, and not all drivers are
> > >> > suited to initialize their devices when they are open. For example,
> > >> > input drivers need to interrogate their devices in order to publish
> > >> > device's capabilities before userspace will open them. When such drivers
> > >> > are compiled into kernel they may stall entire kernel initialization.
> > >> >
> > >> > This change allows drivers request for their probe functions to be
> > >> > called asynchronously during driver and device registration (manual
> > >> > binding is still synchronous). Because async_schedule is used to perform
> > >> > asynchronous calls module loading will still wait for the probing to
> > >> > complete.
> > >> >
> > >> > Note that the end goal is to make the probing asynchronous by default,
> > >> > so annotating drivers with PROBE_PREFER_ASYNCHRONOUS is a temporary
> > >> > measure that allows us to speed up boot process while we validating and
> > >> > fixing the rest of the drivers and preparing userspace.
> > >> >
> > >> > This change is based on earlier patch by "Luis R. Rodriguez"
> > >> > <mcgrof@suse.com>
> > >> >
> > >> > Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
> > >> > ---
> > >> >  drivers/base/base.h    |   1 +
> > >> >  drivers/base/bus.c     |  31 +++++++---
> > >> >  drivers/base/dd.c      | 149 ++++++++++++++++++++++++++++++++++++++++++-------
> > >> >  include/linux/device.h |  28 ++++++++++
> > >> >  4 files changed, 182 insertions(+), 27 deletions(-)
> > >>
> > >> Just noticed this patch.  It caught my eye because I had a hard time
> > >> getting an open coded implementation of asynchronous probing to work
> > >> in the new libnvdimm subsystem.  Especially the messy races of tearing
> > >> things down while probing is still in flight.  I ended up implementing
> > >> asynchronous device registration which eliminated a lot of complexity
> > >> and of course the bugs.  In general I tend to think that async
> > >> registration is less risky than async probe since it keeps wider
> > >> portions of the traditional device model synchronous
> > >
> > > but its not see -DEFER_PROBE even before async probe.
> > 
> > Except in that case you know probe has been seen by the driver at
> > least once.  So I see that as less of a surprise, but point taken.
> > 
> > >> and leverages the
> > >> fact that the device model is already well prepared for asynchronous
> > >> arrival of devices due to hotplug.
> > >
> > > I think this sounds reasonable, do you have your code upstream or posted?
> > 
> > Yes, see nd_device_register() in drivers/nvdimm/bus.c
> 
> It should be I think rather easy for Dmitry to see if he can convert this input
> driver (not yet upstream) to this API and see if the same issues are fixed.

No, I would rather not as it means we lose error handling on device
registration.

Thanks.

-- 
Dmitry

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Dan Williams]

On Mon, Jul 6, 2015 at 4:40 PM, Dmitry Torokhov
<dmitry.torokhov@gmail.com> wrote:
> On Tue, Jul 07, 2015 at 01:23:15AM +0200, Luis R. Rodriguez wrote:
>> On Sat, Jul 04, 2015 at 07:09:19AM -0700, Dan Williams wrote:
>> > On Fri, Jul 3, 2015 at 11:30 AM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
>> > > On Sat, Jun 27, 2015 at 04:45:25PM -0700, Dan Williams wrote:
>> > >> On Mon, Mar 30, 2015 at 4:20 PM, Dmitry Torokhov
>> > >> <dmitry.torokhov@gmail.com> wrote:
>> > >> > Some devices take a long time when initializing, and not all drivers are
>> > >> > suited to initialize their devices when they are open. For example,
>> > >> > input drivers need to interrogate their devices in order to publish
>> > >> > device's capabilities before userspace will open them. When such drivers
>> > >> > are compiled into kernel they may stall entire kernel initialization.
>> > >> >
>> > >> > This change allows drivers request for their probe functions to be
>> > >> > called asynchronously during driver and device registration (manual
>> > >> > binding is still synchronous). Because async_schedule is used to perform
>> > >> > asynchronous calls module loading will still wait for the probing to
>> > >> > complete.
>> > >> >
>> > >> > Note that the end goal is to make the probing asynchronous by default,
>> > >> > so annotating drivers with PROBE_PREFER_ASYNCHRONOUS is a temporary
>> > >> > measure that allows us to speed up boot process while we validating and
>> > >> > fixing the rest of the drivers and preparing userspace.
>> > >> >
>> > >> > This change is based on earlier patch by "Luis R. Rodriguez"
>> > >> > <mcgrof@suse.com>
>> > >> >
>> > >> > Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
>> > >> > ---
>> > >> >  drivers/base/base.h    |   1 +
>> > >> >  drivers/base/bus.c     |  31 +++++++---
>> > >> >  drivers/base/dd.c      | 149 ++++++++++++++++++++++++++++++++++++++++++-------
>> > >> >  include/linux/device.h |  28 ++++++++++
>> > >> >  4 files changed, 182 insertions(+), 27 deletions(-)
>> > >>
>> > >> Just noticed this patch.  It caught my eye because I had a hard time
>> > >> getting an open coded implementation of asynchronous probing to work
>> > >> in the new libnvdimm subsystem.  Especially the messy races of tearing
>> > >> things down while probing is still in flight.  I ended up implementing
>> > >> asynchronous device registration which eliminated a lot of complexity
>> > >> and of course the bugs.  In general I tend to think that async
>> > >> registration is less risky than async probe since it keeps wider
>> > >> portions of the traditional device model synchronous
>> > >
>> > > but its not see -DEFER_PROBE even before async probe.
>> >
>> > Except in that case you know probe has been seen by the driver at
>> > least once.  So I see that as less of a surprise, but point taken.
>> >
>> > >> and leverages the
>> > >> fact that the device model is already well prepared for asynchronous
>> > >> arrival of devices due to hotplug.
>> > >
>> > > I think this sounds reasonable, do you have your code upstream or posted?
>> >
>> > Yes, see nd_device_register() in drivers/nvdimm/bus.c
>>
>> It should be I think rather easy for Dmitry to see if he can convert this input
>> driver (not yet upstream) to this API and see if the same issues are fixed.
>
> No, I would rather not as it means we lose error handling on device
> registration.
>

I think this is a red herring as I don't see how async probing is any
better at handling device registration errors.  The error is logged
and "handled" by the fact that a device fails to appear, what other
action would you take?  In fact libnvdimm does detect registration
failures and reports that in a parent device attribute (at least for a
region device and their namespace child devices).

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Dmitry Torokhov]

On Wed, Jul 08, 2015 at 05:36:04PM -0700, Dan Williams wrote:
> On Mon, Jul 6, 2015 at 4:40 PM, Dmitry Torokhov
> <dmitry.torokhov@gmail.com> wrote:
> > On Tue, Jul 07, 2015 at 01:23:15AM +0200, Luis R. Rodriguez wrote:
> >> On Sat, Jul 04, 2015 at 07:09:19AM -0700, Dan Williams wrote:
> >> > On Fri, Jul 3, 2015 at 11:30 AM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> >> > > On Sat, Jun 27, 2015 at 04:45:25PM -0700, Dan Williams wrote:
> >> > >> On Mon, Mar 30, 2015 at 4:20 PM, Dmitry Torokhov
> >> > >> <dmitry.torokhov@gmail.com> wrote:
> >> > >> > Some devices take a long time when initializing, and not all drivers are
> >> > >> > suited to initialize their devices when they are open. For example,
> >> > >> > input drivers need to interrogate their devices in order to publish
> >> > >> > device's capabilities before userspace will open them. When such drivers
> >> > >> > are compiled into kernel they may stall entire kernel initialization.
> >> > >> >
> >> > >> > This change allows drivers request for their probe functions to be
> >> > >> > called asynchronously during driver and device registration (manual
> >> > >> > binding is still synchronous). Because async_schedule is used to perform
> >> > >> > asynchronous calls module loading will still wait for the probing to
> >> > >> > complete.
> >> > >> >
> >> > >> > Note that the end goal is to make the probing asynchronous by default,
> >> > >> > so annotating drivers with PROBE_PREFER_ASYNCHRONOUS is a temporary
> >> > >> > measure that allows us to speed up boot process while we validating and
> >> > >> > fixing the rest of the drivers and preparing userspace.
> >> > >> >
> >> > >> > This change is based on earlier patch by "Luis R. Rodriguez"
> >> > >> > <mcgrof@suse.com>
> >> > >> >
> >> > >> > Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
> >> > >> > ---
> >> > >> >  drivers/base/base.h    |   1 +
> >> > >> >  drivers/base/bus.c     |  31 +++++++---
> >> > >> >  drivers/base/dd.c      | 149 ++++++++++++++++++++++++++++++++++++++++++-------
> >> > >> >  include/linux/device.h |  28 ++++++++++
> >> > >> >  4 files changed, 182 insertions(+), 27 deletions(-)
> >> > >>
> >> > >> Just noticed this patch.  It caught my eye because I had a hard time
> >> > >> getting an open coded implementation of asynchronous probing to work
> >> > >> in the new libnvdimm subsystem.  Especially the messy races of tearing
> >> > >> things down while probing is still in flight.  I ended up implementing
> >> > >> asynchronous device registration which eliminated a lot of complexity
> >> > >> and of course the bugs.  In general I tend to think that async
> >> > >> registration is less risky than async probe since it keeps wider
> >> > >> portions of the traditional device model synchronous
> >> > >
> >> > > but its not see -DEFER_PROBE even before async probe.
> >> >
> >> > Except in that case you know probe has been seen by the driver at
> >> > least once.  So I see that as less of a surprise, but point taken.
> >> >
> >> > >> and leverages the
> >> > >> fact that the device model is already well prepared for asynchronous
> >> > >> arrival of devices due to hotplug.
> >> > >
> >> > > I think this sounds reasonable, do you have your code upstream or posted?
> >> >
> >> > Yes, see nd_device_register() in drivers/nvdimm/bus.c
> >>
> >> It should be I think rather easy for Dmitry to see if he can convert this input
> >> driver (not yet upstream) to this API and see if the same issues are fixed.
> >
> > No, I would rather not as it means we lose error handling on device
> > registration.
> >
> 
> I think this is a red herring as I don't see how async probing is any
> better at handling device registration errors.  The error is logged
> and "handled" by the fact that a device fails to appear, what other
> action would you take?  In fact libnvdimm does detect registration
> failures and reports that in a parent device attribute (at least for a
> region device and their namespace child devices).

What is libnvdimm behavior if you try to unload a module that tries to
register a device but it failed? Memory leak or crash, right?

-- 
Dmitry

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Dan Williams]

On Wed, Jul 8, 2015 at 5:49 PM, Dmitry Torokhov
<dmitry.torokhov@gmail.com> wrote:
> On Wed, Jul 08, 2015 at 05:36:04PM -0700, Dan Williams wrote:
>> On Mon, Jul 6, 2015 at 4:40 PM, Dmitry Torokhov
>> <dmitry.torokhov@gmail.com> wrote:
>> > On Tue, Jul 07, 2015 at 01:23:15AM +0200, Luis R. Rodriguez wrote:
>> >> On Sat, Jul 04, 2015 at 07:09:19AM -0700, Dan Williams wrote:
>> >> > On Fri, Jul 3, 2015 at 11:30 AM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
>> >> > > On Sat, Jun 27, 2015 at 04:45:25PM -0700, Dan Williams wrote:
>> >> > >> On Mon, Mar 30, 2015 at 4:20 PM, Dmitry Torokhov
>> >> > >> <dmitry.torokhov@gmail.com> wrote:
>> >> > >> > Some devices take a long time when initializing, and not all drivers are
>> >> > >> > suited to initialize their devices when they are open. For example,
>> >> > >> > input drivers need to interrogate their devices in order to publish
>> >> > >> > device's capabilities before userspace will open them. When such drivers
>> >> > >> > are compiled into kernel they may stall entire kernel initialization.
>> >> > >> >
>> >> > >> > This change allows drivers request for their probe functions to be
>> >> > >> > called asynchronously during driver and device registration (manual
>> >> > >> > binding is still synchronous). Because async_schedule is used to perform
>> >> > >> > asynchronous calls module loading will still wait for the probing to
>> >> > >> > complete.
>> >> > >> >
>> >> > >> > Note that the end goal is to make the probing asynchronous by default,
>> >> > >> > so annotating drivers with PROBE_PREFER_ASYNCHRONOUS is a temporary
>> >> > >> > measure that allows us to speed up boot process while we validating and
>> >> > >> > fixing the rest of the drivers and preparing userspace.
>> >> > >> >
>> >> > >> > This change is based on earlier patch by "Luis R. Rodriguez"
>> >> > >> > <mcgrof@suse.com>
>> >> > >> >
>> >> > >> > Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
>> >> > >> > ---
>> >> > >> >  drivers/base/base.h    |   1 +
>> >> > >> >  drivers/base/bus.c     |  31 +++++++---
>> >> > >> >  drivers/base/dd.c      | 149 ++++++++++++++++++++++++++++++++++++++++++-------
>> >> > >> >  include/linux/device.h |  28 ++++++++++
>> >> > >> >  4 files changed, 182 insertions(+), 27 deletions(-)
>> >> > >>
>> >> > >> Just noticed this patch.  It caught my eye because I had a hard time
>> >> > >> getting an open coded implementation of asynchronous probing to work
>> >> > >> in the new libnvdimm subsystem.  Especially the messy races of tearing
>> >> > >> things down while probing is still in flight.  I ended up implementing
>> >> > >> asynchronous device registration which eliminated a lot of complexity
>> >> > >> and of course the bugs.  In general I tend to think that async
>> >> > >> registration is less risky than async probe since it keeps wider
>> >> > >> portions of the traditional device model synchronous
>> >> > >
>> >> > > but its not see -DEFER_PROBE even before async probe.
>> >> >
>> >> > Except in that case you know probe has been seen by the driver at
>> >> > least once.  So I see that as less of a surprise, but point taken.
>> >> >
>> >> > >> and leverages the
>> >> > >> fact that the device model is already well prepared for asynchronous
>> >> > >> arrival of devices due to hotplug.
>> >> > >
>> >> > > I think this sounds reasonable, do you have your code upstream or posted?
>> >> >
>> >> > Yes, see nd_device_register() in drivers/nvdimm/bus.c
>> >>
>> >> It should be I think rather easy for Dmitry to see if he can convert this input
>> >> driver (not yet upstream) to this API and see if the same issues are fixed.
>> >
>> > No, I would rather not as it means we lose error handling on device
>> > registration.
>> >
>>
>> I think this is a red herring as I don't see how async probing is any
>> better at handling device registration errors.  The error is logged
>> and "handled" by the fact that a device fails to appear, what other
>> action would you take?  In fact libnvdimm does detect registration
>> failures and reports that in a parent device attribute (at least for a
>> region device and their namespace child devices).
>
> What is libnvdimm behavior if you try to unload a module that tries to
> register a device but it failed? Memory leak or crash, right?

No, in the case of the "region" driver it is part of the core
libnvdimm and it is pinned while any region device is active.  But, it
is a fair point for a generic facility it would need to consider cases
where a driver registers children and then is unloaded.  I'll need to
think about that one more relative to async probing.

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Dmitry Torokhov]

On Wed, Jul 08, 2015 at 06:00:41PM -0700, Dan Williams wrote:
> On Wed, Jul 8, 2015 at 5:49 PM, Dmitry Torokhov
> <dmitry.torokhov@gmail.com> wrote:
> > On Wed, Jul 08, 2015 at 05:36:04PM -0700, Dan Williams wrote:
> >> On Mon, Jul 6, 2015 at 4:40 PM, Dmitry Torokhov
> >> <dmitry.torokhov@gmail.com> wrote:
> >> > On Tue, Jul 07, 2015 at 01:23:15AM +0200, Luis R. Rodriguez wrote:
> >> >> On Sat, Jul 04, 2015 at 07:09:19AM -0700, Dan Williams wrote:
> >> >> > On Fri, Jul 3, 2015 at 11:30 AM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> >> >> > > On Sat, Jun 27, 2015 at 04:45:25PM -0700, Dan Williams wrote:
> >> >> > >> On Mon, Mar 30, 2015 at 4:20 PM, Dmitry Torokhov
> >> >> > >> <dmitry.torokhov@gmail.com> wrote:
> >> >> > >> > Some devices take a long time when initializing, and not all drivers are
> >> >> > >> > suited to initialize their devices when they are open. For example,
> >> >> > >> > input drivers need to interrogate their devices in order to publish
> >> >> > >> > device's capabilities before userspace will open them. When such drivers
> >> >> > >> > are compiled into kernel they may stall entire kernel initialization.
> >> >> > >> >
> >> >> > >> > This change allows drivers request for their probe functions to be
> >> >> > >> > called asynchronously during driver and device registration (manual
> >> >> > >> > binding is still synchronous). Because async_schedule is used to perform
> >> >> > >> > asynchronous calls module loading will still wait for the probing to
> >> >> > >> > complete.
> >> >> > >> >
> >> >> > >> > Note that the end goal is to make the probing asynchronous by default,
> >> >> > >> > so annotating drivers with PROBE_PREFER_ASYNCHRONOUS is a temporary
> >> >> > >> > measure that allows us to speed up boot process while we validating and
> >> >> > >> > fixing the rest of the drivers and preparing userspace.
> >> >> > >> >
> >> >> > >> > This change is based on earlier patch by "Luis R. Rodriguez"
> >> >> > >> > <mcgrof@suse.com>
> >> >> > >> >
> >> >> > >> > Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
> >> >> > >> > ---
> >> >> > >> >  drivers/base/base.h    |   1 +
> >> >> > >> >  drivers/base/bus.c     |  31 +++++++---
> >> >> > >> >  drivers/base/dd.c      | 149 ++++++++++++++++++++++++++++++++++++++++++-------
> >> >> > >> >  include/linux/device.h |  28 ++++++++++
> >> >> > >> >  4 files changed, 182 insertions(+), 27 deletions(-)
> >> >> > >>
> >> >> > >> Just noticed this patch.  It caught my eye because I had a hard time
> >> >> > >> getting an open coded implementation of asynchronous probing to work
> >> >> > >> in the new libnvdimm subsystem.  Especially the messy races of tearing
> >> >> > >> things down while probing is still in flight.  I ended up implementing
> >> >> > >> asynchronous device registration which eliminated a lot of complexity
> >> >> > >> and of course the bugs.  In general I tend to think that async
> >> >> > >> registration is less risky than async probe since it keeps wider
> >> >> > >> portions of the traditional device model synchronous
> >> >> > >
> >> >> > > but its not see -DEFER_PROBE even before async probe.
> >> >> >
> >> >> > Except in that case you know probe has been seen by the driver at
> >> >> > least once.  So I see that as less of a surprise, but point taken.
> >> >> >
> >> >> > >> and leverages the
> >> >> > >> fact that the device model is already well prepared for asynchronous
> >> >> > >> arrival of devices due to hotplug.
> >> >> > >
> >> >> > > I think this sounds reasonable, do you have your code upstream or posted?
> >> >> >
> >> >> > Yes, see nd_device_register() in drivers/nvdimm/bus.c
> >> >>
> >> >> It should be I think rather easy for Dmitry to see if he can convert this input
> >> >> driver (not yet upstream) to this API and see if the same issues are fixed.
> >> >
> >> > No, I would rather not as it means we lose error handling on device
> >> > registration.
> >> >
> >>
> >> I think this is a red herring as I don't see how async probing is any
> >> better at handling device registration errors.  The error is logged
> >> and "handled" by the fact that a device fails to appear, what other
> >> action would you take?  In fact libnvdimm does detect registration
> >> failures and reports that in a parent device attribute (at least for a
> >> region device and their namespace child devices).
> >
> > What is libnvdimm behavior if you try to unload a module that tries to
> > register a device but it failed? Memory leak or crash, right?
> 
> No, in the case of the "region" driver it is part of the core
> libnvdimm and it is pinned while any region device is active.

No, not quite. Let's take a look for example at nd_btt_probe(). It calls
__nd_btt_create() which in turn calls __nd_device_register() which
returns void and asynchronously schedules device registration. Now
consider the device registration fails. The async code will drop 2
references to the device, effectively freeing it. In the mean time
nd_btt_probe() stores the device pointer which may or may no longer be
valid and goes on it's merry way using it.

The similar thing in nvdimm_create which returns a pointer that may no
longer be valid. I have not traced enough through the code to make sure
if it can blow up, but this kind of situation is not desirable,
especially if the async registration pattern is applied generally
throughout the kernel.

Thanks.

-- 
Dmitry

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Dan Williams]

On Wed, Jul 8, 2015 at 9:44 PM, Dmitry Torokhov
<dmitry.torokhov@gmail.com> wrote:
> On Wed, Jul 08, 2015 at 06:00:41PM -0700, Dan Williams wrote:
>> On Wed, Jul 8, 2015 at 5:49 PM, Dmitry Torokhov
>> <dmitry.torokhov@gmail.com> wrote:
>> > On Wed, Jul 08, 2015 at 05:36:04PM -0700, Dan Williams wrote:
>> >> On Mon, Jul 6, 2015 at 4:40 PM, Dmitry Torokhov
>> >> <dmitry.torokhov@gmail.com> wrote:
>> >> > On Tue, Jul 07, 2015 at 01:23:15AM +0200, Luis R. Rodriguez wrote:
>> >> >> On Sat, Jul 04, 2015 at 07:09:19AM -0700, Dan Williams wrote:
>> >> >> > On Fri, Jul 3, 2015 at 11:30 AM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
>> >> >> > > On Sat, Jun 27, 2015 at 04:45:25PM -0700, Dan Williams wrote:
>> >> >> > >> On Mon, Mar 30, 2015 at 4:20 PM, Dmitry Torokhov
>> >> >> > >> <dmitry.torokhov@gmail.com> wrote:
>> >> >> > >> > Some devices take a long time when initializing, and not all drivers are
>> >> >> > >> > suited to initialize their devices when they are open. For example,
>> >> >> > >> > input drivers need to interrogate their devices in order to publish
>> >> >> > >> > device's capabilities before userspace will open them. When such drivers
>> >> >> > >> > are compiled into kernel they may stall entire kernel initialization.
>> >> >> > >> >
>> >> >> > >> > This change allows drivers request for their probe functions to be
>> >> >> > >> > called asynchronously during driver and device registration (manual
>> >> >> > >> > binding is still synchronous). Because async_schedule is used to perform
>> >> >> > >> > asynchronous calls module loading will still wait for the probing to
>> >> >> > >> > complete.
>> >> >> > >> >
>> >> >> > >> > Note that the end goal is to make the probing asynchronous by default,
>> >> >> > >> > so annotating drivers with PROBE_PREFER_ASYNCHRONOUS is a temporary
>> >> >> > >> > measure that allows us to speed up boot process while we validating and
>> >> >> > >> > fixing the rest of the drivers and preparing userspace.
>> >> >> > >> >
>> >> >> > >> > This change is based on earlier patch by "Luis R. Rodriguez"
>> >> >> > >> > <mcgrof@suse.com>
>> >> >> > >> >
>> >> >> > >> > Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
>> >> >> > >> > ---
>> >> >> > >> >  drivers/base/base.h    |   1 +
>> >> >> > >> >  drivers/base/bus.c     |  31 +++++++---
>> >> >> > >> >  drivers/base/dd.c      | 149 ++++++++++++++++++++++++++++++++++++++++++-------
>> >> >> > >> >  include/linux/device.h |  28 ++++++++++
>> >> >> > >> >  4 files changed, 182 insertions(+), 27 deletions(-)
>> >> >> > >>
>> >> >> > >> Just noticed this patch.  It caught my eye because I had a hard time
>> >> >> > >> getting an open coded implementation of asynchronous probing to work
>> >> >> > >> in the new libnvdimm subsystem.  Especially the messy races of tearing
>> >> >> > >> things down while probing is still in flight.  I ended up implementing
>> >> >> > >> asynchronous device registration which eliminated a lot of complexity
>> >> >> > >> and of course the bugs.  In general I tend to think that async
>> >> >> > >> registration is less risky than async probe since it keeps wider
>> >> >> > >> portions of the traditional device model synchronous
>> >> >> > >
>> >> >> > > but its not see -DEFER_PROBE even before async probe.
>> >> >> >
>> >> >> > Except in that case you know probe has been seen by the driver at
>> >> >> > least once.  So I see that as less of a surprise, but point taken.
>> >> >> >
>> >> >> > >> and leverages the
>> >> >> > >> fact that the device model is already well prepared for asynchronous
>> >> >> > >> arrival of devices due to hotplug.
>> >> >> > >
>> >> >> > > I think this sounds reasonable, do you have your code upstream or posted?
>> >> >> >
>> >> >> > Yes, see nd_device_register() in drivers/nvdimm/bus.c
>> >> >>
>> >> >> It should be I think rather easy for Dmitry to see if he can convert this input
>> >> >> driver (not yet upstream) to this API and see if the same issues are fixed.
>> >> >
>> >> > No, I would rather not as it means we lose error handling on device
>> >> > registration.
>> >> >
>> >>
>> >> I think this is a red herring as I don't see how async probing is any
>> >> better at handling device registration errors.  The error is logged
>> >> and "handled" by the fact that a device fails to appear, what other
>> >> action would you take?  In fact libnvdimm does detect registration
>> >> failures and reports that in a parent device attribute (at least for a
>> >> region device and their namespace child devices).
>> >
>> > What is libnvdimm behavior if you try to unload a module that tries to
>> > register a device but it failed? Memory leak or crash, right?
>>
>> No, in the case of the "region" driver it is part of the core
>> libnvdimm and it is pinned while any region device is active.
>
> No, not quite. Let's take a look for example at nd_btt_probe(). It calls
> __nd_btt_create() which in turn calls __nd_device_register() which
> returns void and asynchronously schedules device registration. Now
> consider the device registration fails. The async code will drop 2
> references to the device, effectively freeing it. In the mean time
> nd_btt_probe() stores the device pointer which may or may no longer be
> valid and goes on it's merry way using it.

nd_btt_probe() is the driver probe for the btt device.  If
registration fails then the device is never probed.

> The similar thing in nvdimm_create which returns a pointer that may no
> longer be valid

Exactly, which is why we fail the entire bus probe if any of the
nvdimm devices failed to register.  See nvdimm_bus_check_dimm_count().

> I have not traced enough through the code to make sure
> if it can blow up, but this kind of situation is not desirable,
> especially if the async registration pattern is applied generally
> throughout the kernel.

I do appreciate the review, but I don't think this signals the death
knell for async registration just yet.

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Tom Gundersen]

On Tue, Jul 7, 2015 at 1:23 AM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> On Sat, Jul 04, 2015 at 07:09:19AM -0700, Dan Williams wrote:
>> On Fri, Jul 3, 2015 at 11:30 AM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
>> > On Sat, Jun 27, 2015 at 04:45:25PM -0700, Dan Williams wrote:
>> >> On Mon, Mar 30, 2015 at 4:20 PM, Dmitry Torokhov
>> >> <dmitry.torokhov@gmail.com> wrote:
>> >> > Some devices take a long time when initializing, and not all drivers are
>> >> > suited to initialize their devices when they are open. For example,
>> >> > input drivers need to interrogate their devices in order to publish
>> >> > device's capabilities before userspace will open them. When such drivers
>> >> > are compiled into kernel they may stall entire kernel initialization.
>> >> >
>> >> > This change allows drivers request for their probe functions to be
>> >> > called asynchronously during driver and device registration (manual
>> >> > binding is still synchronous). Because async_schedule is used to perform
>> >> > asynchronous calls module loading will still wait for the probing to
>> >> > complete.
>> >> >
>> >> > Note that the end goal is to make the probing asynchronous by default,
>> >> > so annotating drivers with PROBE_PREFER_ASYNCHRONOUS is a temporary
>> >> > measure that allows us to speed up boot process while we validating and
>> >> > fixing the rest of the drivers and preparing userspace.
>> >> >
>> >> > This change is based on earlier patch by "Luis R. Rodriguez"
>> >> > <mcgrof@suse.com>
>> >> >
>> >> > Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
>> >> > ---
>> >> >  drivers/base/base.h    |   1 +
>> >> >  drivers/base/bus.c     |  31 +++++++---
>> >> >  drivers/base/dd.c      | 149 ++++++++++++++++++++++++++++++++++++++++++-------
>> >> >  include/linux/device.h |  28 ++++++++++
>> >> >  4 files changed, 182 insertions(+), 27 deletions(-)
>> >>
>> >> Just noticed this patch.  It caught my eye because I had a hard time
>> >> getting an open coded implementation of asynchronous probing to work
>> >> in the new libnvdimm subsystem.  Especially the messy races of tearing
>> >> things down while probing is still in flight.  I ended up implementing
>> >> asynchronous device registration which eliminated a lot of complexity
>> >> and of course the bugs.  In general I tend to think that async
>> >> registration is less risky than async probe since it keeps wider
>> >> portions of the traditional device model synchronous
>> >
>> > but its not see -DEFER_PROBE even before async probe.
>>
>> Except in that case you know probe has been seen by the driver at
>> least once.  So I see that as less of a surprise, but point taken.
>>
>> >> and leverages the
>> >> fact that the device model is already well prepared for asynchronous
>> >> arrival of devices due to hotplug.
>> >
>> > I think this sounds reasonable, do you have your code upstream or posted?
>>
>> Yes, see nd_device_register() in drivers/nvdimm/bus.c
>
> It should be I think rather easy for Dmitry to see if he can convert this input
> driver (not yet upstream) to this API and see if the same issues are fixed.
> This however does not address systemd's assumption over detachment of module
> load and probe. The inherent problem there was the timeout implemented and
> carried in systemd over the worker that uses modlib to load modules. Upon
> review the code was complex enough already and surely increasing the timeout
> helps but that doesn't address all issues with a general timeout in place.
> At SUSE we ended up not using a timeout for kmod built-in commands. That
> leaves the original timeout purpose in place. The code for async probe was
> not put in the kernel though but since its now upstream we should be able
> to replace that userspace systemd work around with async probe, but systemd
> folks would need to decide what they want to do. For full gory details of
> this refer to:
>
> https://bugzilla.novell.com/show_bug.cgi?id=889297

FTR, this does not appear to be public, so I was not able to see it.

>> > If not will you be at Plumbers?
>>
>> Yes.
>
> Great, Tom, Dmitry, will you be at Plumbers?

Sadly, I won't make plumbers this year.

Cheers,

Tom

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Dmitry Torokhov]

On Sat, Jul 04, 2015 at 07:09:19AM -0700, Dan Williams wrote:
> On Fri, Jul 3, 2015 at 11:30 AM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> > On Sat, Jun 27, 2015 at 04:45:25PM -0700, Dan Williams wrote:
> >> On Mon, Mar 30, 2015 at 4:20 PM, Dmitry Torokhov
> >> <dmitry.torokhov@gmail.com> wrote:
> >> > Some devices take a long time when initializing, and not all drivers are
> >> > suited to initialize their devices when they are open. For example,
> >> > input drivers need to interrogate their devices in order to publish
> >> > device's capabilities before userspace will open them. When such drivers
> >> > are compiled into kernel they may stall entire kernel initialization.
> >> >
> >> > This change allows drivers request for their probe functions to be
> >> > called asynchronously during driver and device registration (manual
> >> > binding is still synchronous). Because async_schedule is used to perform
> >> > asynchronous calls module loading will still wait for the probing to
> >> > complete.
> >> >
> >> > Note that the end goal is to make the probing asynchronous by default,
> >> > so annotating drivers with PROBE_PREFER_ASYNCHRONOUS is a temporary
> >> > measure that allows us to speed up boot process while we validating and
> >> > fixing the rest of the drivers and preparing userspace.
> >> >
> >> > This change is based on earlier patch by "Luis R. Rodriguez"
> >> > <mcgrof@suse.com>
> >> >
> >> > Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
> >> > ---
> >> >  drivers/base/base.h    |   1 +
> >> >  drivers/base/bus.c     |  31 +++++++---
> >> >  drivers/base/dd.c      | 149 ++++++++++++++++++++++++++++++++++++++++++-------
> >> >  include/linux/device.h |  28 ++++++++++
> >> >  4 files changed, 182 insertions(+), 27 deletions(-)
> >>
> >> Just noticed this patch.  It caught my eye because I had a hard time
> >> getting an open coded implementation of asynchronous probing to work
> >> in the new libnvdimm subsystem.  Especially the messy races of tearing
> >> things down while probing is still in flight.  I ended up implementing
> >> asynchronous device registration which eliminated a lot of complexity
> >> and of course the bugs.  In general I tend to think that async
> >> registration is less risky than async probe since it keeps wider
> >> portions of the traditional device model synchronous
> >
> > but its not see -DEFER_PROBE even before async probe.
> 
> Except in that case you know probe has been seen by the driver at
> least once.  So I see that as less of a surprise, but point taken.
> 
> >> and leverages the
> >> fact that the device model is already well prepared for asynchronous
> >> arrival of devices due to hotplug.
> >
> > I think this sounds reasonable, do you have your code upstream or posted?
> 
> Yes, see nd_device_register() in drivers/nvdimm/bus.c

So no error handling whatsoever, as expected...

> 
> > If not will you be at Plumbers?
> 
> Yes.

Me too.

> 
> > Maybe we shoudl talk about this as although
> > ChromeOS already likely already jumped on async probe we should address a
> > way forward and path forward for other distributions and I don't think anyone
> > is looking too much into it. async probe came to Linux for two reasons:
> >
> >  * chromeos wanting it
> >  * an incorrect systemd assumption on how the driver core works
> >
> > So long term we still need to address the systemd approach, are they going
> > to be defaulting now to async probe for all modules? How about for built-ins?
> >
> > We should talk about this and maybe at plumbers.
> >
> >> Splitting the "initial probe" from
> >> the "manual probe" case seems like a recipe for confusion.
> >
> > If you can come up with pros / cons on both strategies it'd be
> > valuable.
> 
> The problem I ran into was needing to remove devices that still had
> yet to be probed and not being able to use registration completion vs
> the device_lock() to effectively synchronize the sub-system.

Why do you need to "synchronize the sub-system"? The asynchronous
probing should be transparent to the driver. Just unregister the device
(or the driver) and driver core will ensure that probe() is not in
flight.

Confused.

-- 
Dmitry

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Dan Williams]

On Mon, Jul 6, 2015 at 4:38 PM, Dmitry Torokhov
<dmitry.torokhov@gmail.com> wrote:
> On Sat, Jul 04, 2015 at 07:09:19AM -0700, Dan Williams wrote:
>> The problem I ran into was needing to remove devices that still had
>> yet to be probed and not being able to use registration completion vs
>> the device_lock() to effectively synchronize the sub-system.
>
> Why do you need to "synchronize the sub-system"? The asynchronous
> probing should be transparent to the driver. Just unregister the device
> (or the driver) and driver core will ensure that probe() is not in
> flight.

Async registration is indeed transparent to the driver.  The primary
need to "flush registration" is the case of "region" devices that
reference a set of NVDIMM devices.  A region device requires all
related NVDIMMs to be active before the region can be enabled.

I'll look into a more concrete example of the tradeoffs between
asynchronous probing vs registration.

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Dmitry Torokhov]

On Wed, Jul 08, 2015 at 05:43:23PM -0700, Dan Williams wrote:
> On Mon, Jul 6, 2015 at 4:38 PM, Dmitry Torokhov
> <dmitry.torokhov@gmail.com> wrote:
> > On Sat, Jul 04, 2015 at 07:09:19AM -0700, Dan Williams wrote:
> >> The problem I ran into was needing to remove devices that still had
> >> yet to be probed and not being able to use registration completion vs
> >> the device_lock() to effectively synchronize the sub-system.
> >
> > Why do you need to "synchronize the sub-system"? The asynchronous
> > probing should be transparent to the driver. Just unregister the device
> > (or the driver) and driver core will ensure that probe() is not in
> > flight.
> 
> Async registration is indeed transparent to the driver.  The primary
> need to "flush registration" is the case of "region" devices that
> reference a set of NVDIMM devices.  A region device requires all
> related NVDIMMs to be active before the region can be enabled.

Sounds like you need to call into the subsystem to let it know that the
device is active and activate region devices when they are ready. Could
be either explicit call or you can try using bus notifiers for
bind/unbind events.

BTW, do you handle bind/unbind via sysfs (everyone forgets about this
mechanism)?

Thanks.

-- 
Dmitry

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Dan Williams]

On Wed, Jul 8, 2015 at 5:52 PM, Dmitry Torokhov
<dmitry.torokhov@gmail.com> wrote:
> On Wed, Jul 08, 2015 at 05:43:23PM -0700, Dan Williams wrote:
>> On Mon, Jul 6, 2015 at 4:38 PM, Dmitry Torokhov
>> <dmitry.torokhov@gmail.com> wrote:
>> > On Sat, Jul 04, 2015 at 07:09:19AM -0700, Dan Williams wrote:
>> >> The problem I ran into was needing to remove devices that still had
>> >> yet to be probed and not being able to use registration completion vs
>> >> the device_lock() to effectively synchronize the sub-system.
>> >
>> > Why do you need to "synchronize the sub-system"? The asynchronous
>> > probing should be transparent to the driver. Just unregister the device
>> > (or the driver) and driver core will ensure that probe() is not in
>> > flight.
>>
>> Async registration is indeed transparent to the driver.  The primary
>> need to "flush registration" is the case of "region" devices that
>> reference a set of NVDIMM devices.  A region device requires all
>> related NVDIMMs to be active before the region can be enabled.
>
> Sounds like you need to call into the subsystem to let it know that the
> device is active and activate region devices when they are ready. Could
> be either explicit call or you can try using bus notifiers for
> bind/unbind events.
>
> BTW, do you handle bind/unbind via sysfs (everyone forgets about this
> mechanism)?

bind/unbind via systs is central to how libnvdimm operates.  It's
covered by our unit tests.

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Dmitry Torokhov]

On Wed, Jul 08, 2015 at 05:54:26PM -0700, Dan Williams wrote:
> On Wed, Jul 8, 2015 at 5:52 PM, Dmitry Torokhov
> <dmitry.torokhov@gmail.com> wrote:
> > On Wed, Jul 08, 2015 at 05:43:23PM -0700, Dan Williams wrote:
> >> On Mon, Jul 6, 2015 at 4:38 PM, Dmitry Torokhov
> >> <dmitry.torokhov@gmail.com> wrote:
> >> > On Sat, Jul 04, 2015 at 07:09:19AM -0700, Dan Williams wrote:
> >> >> The problem I ran into was needing to remove devices that still had
> >> >> yet to be probed and not being able to use registration completion vs
> >> >> the device_lock() to effectively synchronize the sub-system.
> >> >
> >> > Why do you need to "synchronize the sub-system"? The asynchronous
> >> > probing should be transparent to the driver. Just unregister the device
> >> > (or the driver) and driver core will ensure that probe() is not in
> >> > flight.
> >>
> >> Async registration is indeed transparent to the driver.  The primary
> >> need to "flush registration" is the case of "region" devices that
> >> reference a set of NVDIMM devices.  A region device requires all
> >> related NVDIMMs to be active before the region can be enabled.
> >
> > Sounds like you need to call into the subsystem to let it know that the
> > device is active and activate region devices when they are ready. Could
> > be either explicit call or you can try using bus notifiers for
> > bind/unbind events.
> >
> > BTW, do you handle bind/unbind via sysfs (everyone forgets about this
> > mechanism)?
> 
> bind/unbind via systs is central to how libnvdimm operates.  It's
> covered by our unit tests.

Ah, excellent then.

-- 
Dmitry

[PATCH v2 0/8] Asynchronous device/driver probing support

[Dmitry Torokhov]

This series is a combination of changes proposed by Luis a couple months
ago and implementation used by Chrome OS. The issue we are trying to solve
here is "slow" devices and drivers spending "too much time" in their probe()
methods and it affects:

- overall kernel boot process when drivers are compiled into the kernel
  and slow devices stall entire boot progress;
- systemd desire to time out module loading process.

Unlike Luis' proposal we do make use of asycn_schedule() infrastructure
instead of using a dedicated workqueue, so all  existing synchronization
points in kernel that wait for device registration still work the same.
Also, the asynchronous probing is done not only during driver registration
(i.e. when devices are probed asynchronously only if they are registered
before the driver), but also during device registration and deferred probe
handling. This way slow devices do not stall kernel boot even when drivers
are compiled into the kernel.

The last patch is for adventurous people to try and force
fully-asynchronous boot. It works for me with limited success - I can boot
Rockhip-based box to userspace as long as I force serial to be sychronously
probed and ignore the fact that most devices are using "dummy" regulators
as regulator subsystem really expects regulators to be registered in
orderly fashion on OF-based systems.

Changes from v1:

- Changed verbage in change logs and code to emphasise that
  PROBE_PREFER_ASYNCHRONOUS is a temporary measure and the end goal is
  to enable asynchronous probing by default, as requested by Tejun.

Thanks,
Dmitry

[PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Dmitry Torokhov]

Some devices take a long time when initializing, and not all drivers are
suited to initialize their devices when they are open. For example,
input drivers need to interrogate their devices in order to publish
device's capabilities before userspace will open them. When such drivers
are compiled into kernel they may stall entire kernel initialization.

This change allows drivers request for their probe functions to be
called asynchronously during driver and device registration (manual
binding is still synchronous). Because async_schedule is used to perform
asynchronous calls module loading will still wait for the probing to
complete.

Note that the end goal is to make the probing asynchronous by default,
so annotating drivers with PROBE_PREFER_ASYNCHRONOUS is a temporary
measure that allows us to speed up boot process while we validating and
fixing the rest of the drivers and preparing userspace.

This change is based on earlier patch by "Luis R. Rodriguez"
<mcgrof@suse.com>

Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
---
 drivers/base/base.h    |   1 +
 drivers/base/bus.c     |  31 +++++++---
 drivers/base/dd.c      | 149 ++++++++++++++++++++++++++++++++++++++++++-------
 include/linux/device.h |  28 ++++++++++
 4 files changed, 182 insertions(+), 27 deletions(-)

diff --git a/drivers/base/base.h b/drivers/base/base.h
index 251c5d3..fd3347d 100644
--- a/drivers/base/base.h
+++ b/drivers/base/base.h
@@ -116,6 +116,7 @@ static inline int driver_match_device(struct device_driver *drv,
 {
 	return drv->bus->match ? drv->bus->match(dev, drv) : 1;
 }
+extern bool driver_allows_async_probing(struct device_driver *drv);
 
 extern int driver_add_groups(struct device_driver *drv,
 			     const struct attribute_group **groups);
diff --git a/drivers/base/bus.c b/drivers/base/bus.c
index 79bc203..5005924 100644
--- a/drivers/base/bus.c
+++ b/drivers/base/bus.c
@@ -10,6 +10,7 @@
  *
  */
 
+#include <linux/async.h>
 #include <linux/device.h>
 #include <linux/module.h>
 #include <linux/errno.h>
@@ -549,15 +550,12 @@ void bus_probe_device(struct device *dev)
 {
 	struct bus_type *bus = dev->bus;
 	struct subsys_interface *sif;
-	int ret;
 
 	if (!bus)
 		return;
 
-	if (bus->p->drivers_autoprobe) {
-		ret = device_attach(dev);
-		WARN_ON(ret < 0);
-	}
+	if (bus->p->drivers_autoprobe)
+		device_initial_probe(dev);
 
 	mutex_lock(&bus->p->mutex);
 	list_for_each_entry(sif, &bus->p->interfaces, node)
@@ -659,6 +657,17 @@ static ssize_t uevent_store(struct device_driver *drv, const char *buf,
 }
 static DRIVER_ATTR_WO(uevent);
 
+static void driver_attach_async(void *_drv, async_cookie_t cookie)
+{
+	struct device_driver *drv = _drv;
+	int ret;
+
+	ret = driver_attach(drv);
+
+	pr_debug("bus: '%s': driver %s async attach completed: %d\n",
+		 drv->bus->name, drv->name, ret);
+}
+
 /**
  * bus_add_driver - Add a driver to the bus.
  * @drv: driver.
@@ -691,9 +700,15 @@ int bus_add_driver(struct device_driver *drv)
 
 	klist_add_tail(&priv->knode_bus, &bus->p->klist_drivers);
 	if (drv->bus->p->drivers_autoprobe) {
-		error = driver_attach(drv);
-		if (error)
-			goto out_unregister;
+		if (driver_allows_async_probing(drv)) {
+			pr_debug("bus: '%s': probing driver %s asynchronously\n",
+				drv->bus->name, drv->name);
+			async_schedule(driver_attach_async, drv);
+		} else {
+			error = driver_attach(drv);
+			if (error)
+				goto out_unregister;
+		}
 	}
 	module_add_driver(drv->owner, drv);
 
diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index e843fdb..2ad33b2 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -417,31 +417,95 @@ int driver_probe_device(struct device_driver *drv, struct device *dev)
 	return ret;
 }
 
-static int __device_attach(struct device_driver *drv, void *data)
+bool driver_allows_async_probing(struct device_driver *drv)
 {
-	struct device *dev = data;
+	return drv->probe_type == PROBE_PREFER_ASYNCHRONOUS;
+}
+
+struct device_attach_data {
+	struct device *dev;
+
+	/*
+	 * Indicates whether we are are considering asynchronous probing or
+	 * not. Only initial binding after device or driver registration
+	 * (including deferral processing) may be done asynchronously, the
+	 * rest is always synchronous, as we expect it is being done by
+	 * request from userspace.
+	 */
+	bool check_async;
+
+	/*
+	 * Indicates if we are binding synchronous or asynchronous drivers.
+	 * When asynchronous probing is enabled we'll execute 2 passes
+	 * over drivers: first pass doing synchronous probing and second
+	 * doing asynchronous probing (if synchronous did not succeed -
+	 * most likely because there was no driver requiring synchronous
+	 * probing - and we found asynchronous driver during first pass).
+	 * The 2 passes are done because we can't shoot asynchronous
+	 * probe for given device and driver from bus_for_each_drv() since
+	 * driver pointer is not guaranteed to stay valid once
+	 * bus_for_each_drv() iterates to the next driver on the bus.
+	 */
+	bool want_async;
+
+	/*
+	 * We'll set have_async to 'true' if, while scanning for matching
+	 * driver, we'll encounter one that requests asynchronous probing.
+	 */
+	bool have_async;
+};
+
+static int __device_attach_driver(struct device_driver *drv, void *_data)
+{
+	struct device_attach_data *data = _data;
+	struct device *dev = data->dev;
+	bool async_allowed;
+
+	/*
+	 * Check if device has already been claimed. This may
+	 * happen with driver loading, device discovery/registration,
+	 * and deferred probe processing happens all at once with
+	 * multiple threads.
+	 */
+	if (dev->driver)
+		return -EBUSY;
 
 	if (!driver_match_device(drv, dev))
 		return 0;
 
+	async_allowed = driver_allows_async_probing(drv);
+
+	if (async_allowed)
+		data->have_async = true;
+
+	if (data->check_async && async_allowed != data->want_async)
+		return 0;
+
 	return driver_probe_device(drv, dev);
 }
 
-/**
- * device_attach - try to attach device to a driver.
- * @dev: device.
- *
- * Walk the list of drivers that the bus has and call
- * driver_probe_device() for each pair. If a compatible
- * pair is found, break out and return.
- *
- * Returns 1 if the device was bound to a driver;
- * 0 if no matching driver was found;
- * -ENODEV if the device is not registered.
- *
- * When called for a USB interface, @dev->parent lock must be held.
- */
-int device_attach(struct device *dev)
+static void __device_attach_async_helper(void *_dev, async_cookie_t cookie)
+{
+	struct device *dev = _dev;
+	struct device_attach_data data = {
+		.dev		= dev,
+		.check_async	= true,
+		.want_async	= true,
+	};
+
+	device_lock(dev);
+
+	bus_for_each_drv(dev->bus, NULL, &data, __device_attach_driver);
+	dev_dbg(dev, "async probe completed\n");
+
+	pm_request_idle(dev);
+
+	device_unlock(dev);
+
+	put_device(dev);
+}
+
+int __device_attach(struct device *dev, bool allow_async)
 {
 	int ret = 0;
 
@@ -459,15 +523,59 @@ int device_attach(struct device *dev)
 			ret = 0;
 		}
 	} else {
-		ret = bus_for_each_drv(dev->bus, NULL, dev, __device_attach);
-		pm_request_idle(dev);
+		struct device_attach_data data = {
+			.dev = dev,
+			.check_async = allow_async,
+			.want_async = false,
+		};
+
+		ret = bus_for_each_drv(dev->bus, NULL, &data,
+					__device_attach_driver);
+		if (!ret && allow_async && data.have_async) {
+			/*
+			 * If we could not find appropriate driver
+			 * synchronously and we are allowed to do
+			 * async probes and there are drivers that
+			 * want to probe asynchronously, we'll
+			 * try them.
+			 */
+			dev_dbg(dev, "scheduling asynchronous probe\n");
+			get_device(dev);
+			async_schedule(__device_attach_async_helper, dev);
+		} else {
+			pm_request_idle(dev);
+		}
 	}
 out_unlock:
 	device_unlock(dev);
 	return ret;
 }
+
+/**
+ * device_attach - try to attach device to a driver.
+ * @dev: device.
+ *
+ * Walk the list of drivers that the bus has and call
+ * driver_probe_device() for each pair. If a compatible
+ * pair is found, break out and return.
+ *
+ * Returns 1 if the device was bound to a driver;
+ * 0 if no matching driver was found;
+ * -ENODEV if the device is not registered.
+ *
+ * When called for a USB interface, @dev->parent lock must be held.
+ */
+int device_attach(struct device *dev)
+{
+	return __device_attach(dev, false);
+}
 EXPORT_SYMBOL_GPL(device_attach);
 
+void device_initial_probe(struct device *dev)
+{
+	__device_attach(dev, true);
+}
+
 static int __driver_attach(struct device *dev, void *data)
 {
 	struct device_driver *drv = data;
@@ -522,6 +630,9 @@ static void __device_release_driver(struct device *dev)
 
 	drv = dev->driver;
 	if (drv) {
+		if (driver_allows_async_probing(drv))
+			async_synchronize_full();
+
 		pm_runtime_get_sync(dev);
 
 		driver_sysfs_remove(dev);
diff --git a/include/linux/device.h b/include/linux/device.h
index 884aa6e..400cacd 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -196,12 +196,38 @@ extern struct kset *bus_get_kset(struct bus_type *bus);
 extern struct klist *bus_get_device_klist(struct bus_type *bus);
 
 /**
+ * enum probe_type - device driver probe type to try
+ *	Device drivers may opt in for special handling of their
+ *	respective probe routines. This tells the core what to
+ *	expect and prefer.
+ *
+ * @PROBE_SYNCHRONOUS: Default. Drivers expect their probe routines
+ *	to run synchronously with driver and device registration
+ *	(with the exception of -EPROBE_DEFER handling - re-probing
+ *	always ends up being done asynchronously).
+ * @PROBE_PREFER_ASYNCHRONOUS: Drivers for "slow" devices which
+ *	probing order is not essential for booting the system may
+ *	opt into executing their probes asynchronously.
+ *
+ * Note that the end goal is to switch the kernel to use asynchronous
+ * probing by default, so annotating drivers with
+ * %PROBE_PREFER_ASYNCHRONOUS is a temporary measure that allows us
+ * to speed up boot process while we are validating the rest of the
+ * drivers.
+ */
+enum probe_type {
+	PROBE_SYNCHRONOUS,
+	PROBE_PREFER_ASYNCHRONOUS,
+};
+
+/**
  * struct device_driver - The basic device driver structure
  * @name:	Name of the device driver.
  * @bus:	The bus which the device of this driver belongs to.
  * @owner:	The module owner.
  * @mod_name:	Used for built-in modules.
  * @suppress_bind_attrs: Disables bind/unbind via sysfs.
+ * @probe_type:	Type of the probe (synchronous or asynchronous) to use.
  * @of_match_table: The open firmware table.
  * @acpi_match_table: The ACPI match table.
  * @probe:	Called to query the existence of a specific device,
@@ -235,6 +261,7 @@ struct device_driver {
 	const char		*mod_name;	/* used for built-in modules */
 
 	bool suppress_bind_attrs;	/* disables bind/unbind via sysfs */
+	enum probe_type probe_type;
 
 	const struct of_device_id	*of_match_table;
 	const struct acpi_device_id	*acpi_match_table;
@@ -972,6 +999,7 @@ extern int __must_check device_bind_driver(struct device *dev);
 extern void device_release_driver(struct device *dev);
 extern int  __must_check device_attach(struct device *dev);
 extern int __must_check driver_attach(struct device_driver *drv);
+extern void device_initial_probe(struct device *dev);
 extern int __must_check device_reprobe(struct device *dev);
 
 /*
-- 
2.2.0.rc0.207.ga3a616c

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Tomeu Vizoso]

On 31 March 2015 at 01:20, Dmitry Torokhov <dmitry.torokhov@gmail.com> wrote:
> Some devices take a long time when initializing, and not all drivers are
> suited to initialize their devices when they are open. For example,
> input drivers need to interrogate their devices in order to publish
> device's capabilities before userspace will open them. When such drivers
> are compiled into kernel they may stall entire kernel initialization.
>
> This change allows drivers request for their probe functions to be
> called asynchronously during driver and device registration (manual
> binding is still synchronous). Because async_schedule is used to perform
> asynchronous calls module loading will still wait for the probing to
> complete.

But what about parents? Don't we need to make sure that before probing
a device its parent has finished probing already? This backtrace
illustrates the problem:

[<c0014818>] (__dabt_svc) from [<c03737ac>] (host1x_syncpt_alloc+0x14/0x134)
[<c03737ac>] (host1x_syncpt_alloc) from [<c03738f4>]
(host1x_syncpt_request+0x28/0x2c)
[<c03738f4>] (host1x_syncpt_request) from [<c03b55ec>]
(tegra_dc_probe+0x198/0x35c)
[<c03b55ec>] (tegra_dc_probe) from [<c03cb5a0>] (platform_drv_probe+0x54/0xbc)
[<c03cb5a0>] (platform_drv_probe) from [<c03c96e0>]
(driver_probe_device+0x184/0x2c4)
[<c03c96e0>] (driver_probe_device) from [<c03c98bc>] (__driver_attach+0x9c/0xa0)
[<c03c98bc>] (__driver_attach) from [<c03c78d8>] (bus_for_each_dev+0x78/0xac)
[<c03c78d8>] (bus_for_each_dev) from [<c03c9070>] (driver_attach+0x2c/0x30)
[<c03c9070>] (driver_attach) from [<c03c7e10>] (driver_attach_async+0x18/0x1c)
[<c03c7e10>] (driver_attach_async) from [<c004afd0>]
(async_run_entry_fn+0x58/0x128)
[<c004afd0>] (async_run_entry_fn) from [<c0042470>]
(process_one_work+0x140/0x50c)
[<c0042470>] (process_one_work) from [<c0042890>] (worker_thread+0x54/0x52c)
[<c0042890>] (worker_thread) from [<c0048554>] (kthread+0xec/0x104)
[<c0048554>] (kthread) from [<c000fc28>] (ret_from_fork+0x14/0x2c)

host1x_syncpt_request() assumes that the parent of the DC (host1x) has
been probed already and its drvdata is available.

Thanks,

Tomeu

> Note that the end goal is to make the probing asynchronous by default,
> so annotating drivers with PROBE_PREFER_ASYNCHRONOUS is a temporary
> measure that allows us to speed up boot process while we validating and
> fixing the rest of the drivers and preparing userspace.
>
> This change is based on earlier patch by "Luis R. Rodriguez"
> <mcgrof@suse.com>
>
> Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
> ---
>  drivers/base/base.h    |   1 +
>  drivers/base/bus.c     |  31 +++++++---
>  drivers/base/dd.c      | 149 ++++++++++++++++++++++++++++++++++++++++++-------
>  include/linux/device.h |  28 ++++++++++
>  4 files changed, 182 insertions(+), 27 deletions(-)
>
> diff --git a/drivers/base/base.h b/drivers/base/base.h
> index 251c5d3..fd3347d 100644
> --- a/drivers/base/base.h
> +++ b/drivers/base/base.h
> @@ -116,6 +116,7 @@ static inline int driver_match_device(struct device_driver *drv,
>  {
>         return drv->bus->match ? drv->bus->match(dev, drv) : 1;
>  }
> +extern bool driver_allows_async_probing(struct device_driver *drv);
>
>  extern int driver_add_groups(struct device_driver *drv,
>                              const struct attribute_group **groups);
> diff --git a/drivers/base/bus.c b/drivers/base/bus.c
> index 79bc203..5005924 100644
> --- a/drivers/base/bus.c
> +++ b/drivers/base/bus.c
> @@ -10,6 +10,7 @@
>   *
>   */
>
> +#include <linux/async.h>
>  #include <linux/device.h>
>  #include <linux/module.h>
>  #include <linux/errno.h>
> @@ -549,15 +550,12 @@ void bus_probe_device(struct device *dev)
>  {
>         struct bus_type *bus = dev->bus;
>         struct subsys_interface *sif;
> -       int ret;
>
>         if (!bus)
>                 return;
>
> -       if (bus->p->drivers_autoprobe) {
> -               ret = device_attach(dev);
> -               WARN_ON(ret < 0);
> -       }
> +       if (bus->p->drivers_autoprobe)
> +               device_initial_probe(dev);
>
>         mutex_lock(&bus->p->mutex);
>         list_for_each_entry(sif, &bus->p->interfaces, node)
> @@ -659,6 +657,17 @@ static ssize_t uevent_store(struct device_driver *drv, const char *buf,
>  }
>  static DRIVER_ATTR_WO(uevent);
>
> +static void driver_attach_async(void *_drv, async_cookie_t cookie)
> +{
> +       struct device_driver *drv = _drv;
> +       int ret;
> +
> +       ret = driver_attach(drv);
> +
> +       pr_debug("bus: '%s': driver %s async attach completed: %d\n",
> +                drv->bus->name, drv->name, ret);
> +}
> +
>  /**
>   * bus_add_driver - Add a driver to the bus.
>   * @drv: driver.
> @@ -691,9 +700,15 @@ int bus_add_driver(struct device_driver *drv)
>
>         klist_add_tail(&priv->knode_bus, &bus->p->klist_drivers);
>         if (drv->bus->p->drivers_autoprobe) {
> -               error = driver_attach(drv);
> -               if (error)
> -                       goto out_unregister;
> +               if (driver_allows_async_probing(drv)) {
> +                       pr_debug("bus: '%s': probing driver %s asynchronously\n",
> +                               drv->bus->name, drv->name);
> +                       async_schedule(driver_attach_async, drv);
> +               } else {
> +                       error = driver_attach(drv);
> +                       if (error)
> +                               goto out_unregister;
> +               }
>         }
>         module_add_driver(drv->owner, drv);
>
> diff --git a/drivers/base/dd.c b/drivers/base/dd.c
> index e843fdb..2ad33b2 100644
> --- a/drivers/base/dd.c
> +++ b/drivers/base/dd.c
> @@ -417,31 +417,95 @@ int driver_probe_device(struct device_driver *drv, struct device *dev)
>         return ret;
>  }
>
> -static int __device_attach(struct device_driver *drv, void *data)
> +bool driver_allows_async_probing(struct device_driver *drv)
>  {
> -       struct device *dev = data;
> +       return drv->probe_type == PROBE_PREFER_ASYNCHRONOUS;
> +}
> +
> +struct device_attach_data {
> +       struct device *dev;
> +
> +       /*
> +        * Indicates whether we are are considering asynchronous probing or
> +        * not. Only initial binding after device or driver registration
> +        * (including deferral processing) may be done asynchronously, the
> +        * rest is always synchronous, as we expect it is being done by
> +        * request from userspace.
> +        */
> +       bool check_async;
> +
> +       /*
> +        * Indicates if we are binding synchronous or asynchronous drivers.
> +        * When asynchronous probing is enabled we'll execute 2 passes
> +        * over drivers: first pass doing synchronous probing and second
> +        * doing asynchronous probing (if synchronous did not succeed -
> +        * most likely because there was no driver requiring synchronous
> +        * probing - and we found asynchronous driver during first pass).
> +        * The 2 passes are done because we can't shoot asynchronous
> +        * probe for given device and driver from bus_for_each_drv() since
> +        * driver pointer is not guaranteed to stay valid once
> +        * bus_for_each_drv() iterates to the next driver on the bus.
> +        */
> +       bool want_async;
> +
> +       /*
> +        * We'll set have_async to 'true' if, while scanning for matching
> +        * driver, we'll encounter one that requests asynchronous probing.
> +        */
> +       bool have_async;
> +};
> +
> +static int __device_attach_driver(struct device_driver *drv, void *_data)
> +{
> +       struct device_attach_data *data = _data;
> +       struct device *dev = data->dev;
> +       bool async_allowed;
> +
> +       /*
> +        * Check if device has already been claimed. This may
> +        * happen with driver loading, device discovery/registration,
> +        * and deferred probe processing happens all at once with
> +        * multiple threads.
> +        */
> +       if (dev->driver)
> +               return -EBUSY;
>
>         if (!driver_match_device(drv, dev))
>                 return 0;
>
> +       async_allowed = driver_allows_async_probing(drv);
> +
> +       if (async_allowed)
> +               data->have_async = true;
> +
> +       if (data->check_async && async_allowed != data->want_async)
> +               return 0;
> +
>         return driver_probe_device(drv, dev);
>  }
>
> -/**
> - * device_attach - try to attach device to a driver.
> - * @dev: device.
> - *
> - * Walk the list of drivers that the bus has and call
> - * driver_probe_device() for each pair. If a compatible
> - * pair is found, break out and return.
> - *
> - * Returns 1 if the device was bound to a driver;
> - * 0 if no matching driver was found;
> - * -ENODEV if the device is not registered.
> - *
> - * When called for a USB interface, @dev->parent lock must be held.
> - */
> -int device_attach(struct device *dev)
> +static void __device_attach_async_helper(void *_dev, async_cookie_t cookie)
> +{
> +       struct device *dev = _dev;
> +       struct device_attach_data data = {
> +               .dev            = dev,
> +               .check_async    = true,
> +               .want_async     = true,
> +       };
> +
> +       device_lock(dev);
> +
> +       bus_for_each_drv(dev->bus, NULL, &data, __device_attach_driver);
> +       dev_dbg(dev, "async probe completed\n");
> +
> +       pm_request_idle(dev);
> +
> +       device_unlock(dev);
> +
> +       put_device(dev);
> +}
> +
> +int __device_attach(struct device *dev, bool allow_async)
>  {
>         int ret = 0;
>
> @@ -459,15 +523,59 @@ int device_attach(struct device *dev)
>                         ret = 0;
>                 }
>         } else {
> -               ret = bus_for_each_drv(dev->bus, NULL, dev, __device_attach);
> -               pm_request_idle(dev);
> +               struct device_attach_data data = {
> +                       .dev = dev,
> +                       .check_async = allow_async,
> +                       .want_async = false,
> +               };
> +
> +               ret = bus_for_each_drv(dev->bus, NULL, &data,
> +                                       __device_attach_driver);
> +               if (!ret && allow_async && data.have_async) {
> +                       /*
> +                        * If we could not find appropriate driver
> +                        * synchronously and we are allowed to do
> +                        * async probes and there are drivers that
> +                        * want to probe asynchronously, we'll
> +                        * try them.
> +                        */
> +                       dev_dbg(dev, "scheduling asynchronous probe\n");
> +                       get_device(dev);
> +                       async_schedule(__device_attach_async_helper, dev);
> +               } else {
> +                       pm_request_idle(dev);
> +               }
>         }
>  out_unlock:
>         device_unlock(dev);
>         return ret;
>  }
> +
> +/**
> + * device_attach - try to attach device to a driver.
> + * @dev: device.
> + *
> + * Walk the list of drivers that the bus has and call
> + * driver_probe_device() for each pair. If a compatible
> + * pair is found, break out and return.
> + *
> + * Returns 1 if the device was bound to a driver;
> + * 0 if no matching driver was found;
> + * -ENODEV if the device is not registered.
> + *
> + * When called for a USB interface, @dev->parent lock must be held.
> + */
> +int device_attach(struct device *dev)
> +{
> +       return __device_attach(dev, false);
> +}
>  EXPORT_SYMBOL_GPL(device_attach);
>
> +void device_initial_probe(struct device *dev)
> +{
> +       __device_attach(dev, true);
> +}
> +
>  static int __driver_attach(struct device *dev, void *data)
>  {
>         struct device_driver *drv = data;
> @@ -522,6 +630,9 @@ static void __device_release_driver(struct device *dev)
>
>         drv = dev->driver;
>         if (drv) {
> +               if (driver_allows_async_probing(drv))
> +                       async_synchronize_full();
> +
>                 pm_runtime_get_sync(dev);
>
>                 driver_sysfs_remove(dev);
> diff --git a/include/linux/device.h b/include/linux/device.h
> index 884aa6e..400cacd 100644
> --- a/include/linux/device.h
> +++ b/include/linux/device.h
> @@ -196,12 +196,38 @@ extern struct kset *bus_get_kset(struct bus_type *bus);
>  extern struct klist *bus_get_device_klist(struct bus_type *bus);
>
>  /**
> + * enum probe_type - device driver probe type to try
> + *     Device drivers may opt in for special handling of their
> + *     respective probe routines. This tells the core what to
> + *     expect and prefer.
> + *
> + * @PROBE_SYNCHRONOUS: Default. Drivers expect their probe routines
> + *     to run synchronously with driver and device registration
> + *     (with the exception of -EPROBE_DEFER handling - re-probing
> + *     always ends up being done asynchronously).
> + * @PROBE_PREFER_ASYNCHRONOUS: Drivers for "slow" devices which
> + *     probing order is not essential for booting the system may
> + *     opt into executing their probes asynchronously.
> + *
> + * Note that the end goal is to switch the kernel to use asynchronous
> + * probing by default, so annotating drivers with
> + * %PROBE_PREFER_ASYNCHRONOUS is a temporary measure that allows us
> + * to speed up boot process while we are validating the rest of the
> + * drivers.
> + */
> +enum probe_type {
> +       PROBE_SYNCHRONOUS,
> +       PROBE_PREFER_ASYNCHRONOUS,
> +};
> +
> +/**
>   * struct device_driver - The basic device driver structure
>   * @name:      Name of the device driver.
>   * @bus:       The bus which the device of this driver belongs to.
>   * @owner:     The module owner.
>   * @mod_name:  Used for built-in modules.
>   * @suppress_bind_attrs: Disables bind/unbind via sysfs.
> + * @probe_type:        Type of the probe (synchronous or asynchronous) to use.
>   * @of_match_table: The open firmware table.
>   * @acpi_match_table: The ACPI match table.
>   * @probe:     Called to query the existence of a specific device,
> @@ -235,6 +261,7 @@ struct device_driver {
>         const char              *mod_name;      /* used for built-in modules */
>
>         bool suppress_bind_attrs;       /* disables bind/unbind via sysfs */
> +       enum probe_type probe_type;
>
>         const struct of_device_id       *of_match_table;
>         const struct acpi_device_id     *acpi_match_table;
> @@ -972,6 +999,7 @@ extern int __must_check device_bind_driver(struct device *dev);
>  extern void device_release_driver(struct device *dev);
>  extern int  __must_check device_attach(struct device *dev);
>  extern int __must_check driver_attach(struct device_driver *drv);
> +extern void device_initial_probe(struct device *dev);
>  extern int __must_check device_reprobe(struct device *dev);
>
>  /*
> --
> 2.2.0.rc0.207.ga3a616c
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Tomeu Vizoso]

On 29 May 2015 at 12:48, Tomeu Vizoso <tomeu.vizoso@gmail.com> wrote:
> On 31 March 2015 at 01:20, Dmitry Torokhov <dmitry.torokhov@gmail.com> wrote:
>> Some devices take a long time when initializing, and not all drivers are
>> suited to initialize their devices when they are open. For example,
>> input drivers need to interrogate their devices in order to publish
>> device's capabilities before userspace will open them. When such drivers
>> are compiled into kernel they may stall entire kernel initialization.
>>
>> This change allows drivers request for their probe functions to be
>> called asynchronously during driver and device registration (manual
>> binding is still synchronous). Because async_schedule is used to perform
>> asynchronous calls module loading will still wait for the probing to
>> complete.
>
> But what about parents? Don't we need to make sure that before probing
> a device its parent has finished probing already?

Have realized that this is an existing problem that was just made more
probable by async probe, as before async probing the parent could have
deferred its probe and then its children would be probed.

Wonder if drivers should be modified to defer its probe if their
parent isn't probed yet, or if we can codify that in dd.c.

Regards,

Tomeu

> This backtrace
> illustrates the problem:
>
> [<c0014818>] (__dabt_svc) from [<c03737ac>] (host1x_syncpt_alloc+0x14/0x134)
> [<c03737ac>] (host1x_syncpt_alloc) from [<c03738f4>]
> (host1x_syncpt_request+0x28/0x2c)
> [<c03738f4>] (host1x_syncpt_request) from [<c03b55ec>]
> (tegra_dc_probe+0x198/0x35c)
> [<c03b55ec>] (tegra_dc_probe) from [<c03cb5a0>] (platform_drv_probe+0x54/0xbc)
> [<c03cb5a0>] (platform_drv_probe) from [<c03c96e0>]
> (driver_probe_device+0x184/0x2c4)
> [<c03c96e0>] (driver_probe_device) from [<c03c98bc>] (__driver_attach+0x9c/0xa0)
> [<c03c98bc>] (__driver_attach) from [<c03c78d8>] (bus_for_each_dev+0x78/0xac)
> [<c03c78d8>] (bus_for_each_dev) from [<c03c9070>] (driver_attach+0x2c/0x30)
> [<c03c9070>] (driver_attach) from [<c03c7e10>] (driver_attach_async+0x18/0x1c)
> [<c03c7e10>] (driver_attach_async) from [<c004afd0>]
> (async_run_entry_fn+0x58/0x128)
> [<c004afd0>] (async_run_entry_fn) from [<c0042470>]
> (process_one_work+0x140/0x50c)
> [<c0042470>] (process_one_work) from [<c0042890>] (worker_thread+0x54/0x52c)
> [<c0042890>] (worker_thread) from [<c0048554>] (kthread+0xec/0x104)
> [<c0048554>] (kthread) from [<c000fc28>] (ret_from_fork+0x14/0x2c)
>
> host1x_syncpt_request() assumes that the parent of the DC (host1x) has
> been probed already and its drvdata is available.
>
> Thanks,
>
> Tomeu
>
>> Note that the end goal is to make the probing asynchronous by default,
>> so annotating drivers with PROBE_PREFER_ASYNCHRONOUS is a temporary
>> measure that allows us to speed up boot process while we validating and
>> fixing the rest of the drivers and preparing userspace.
>>
>> This change is based on earlier patch by "Luis R. Rodriguez"
>> <mcgrof@suse.com>
>>
>> Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
>> ---
>>  drivers/base/base.h    |   1 +
>>  drivers/base/bus.c     |  31 +++++++---
>>  drivers/base/dd.c      | 149 ++++++++++++++++++++++++++++++++++++++++++-------
>>  include/linux/device.h |  28 ++++++++++
>>  4 files changed, 182 insertions(+), 27 deletions(-)
>>
>> diff --git a/drivers/base/base.h b/drivers/base/base.h
>> index 251c5d3..fd3347d 100644
>> --- a/drivers/base/base.h
>> +++ b/drivers/base/base.h
>> @@ -116,6 +116,7 @@ static inline int driver_match_device(struct device_driver *drv,
>>  {
>>         return drv->bus->match ? drv->bus->match(dev, drv) : 1;
>>  }
>> +extern bool driver_allows_async_probing(struct device_driver *drv);
>>
>>  extern int driver_add_groups(struct device_driver *drv,
>>                              const struct attribute_group **groups);
>> diff --git a/drivers/base/bus.c b/drivers/base/bus.c
>> index 79bc203..5005924 100644
>> --- a/drivers/base/bus.c
>> +++ b/drivers/base/bus.c
>> @@ -10,6 +10,7 @@
>>   *
>>   */
>>
>> +#include <linux/async.h>
>>  #include <linux/device.h>
>>  #include <linux/module.h>
>>  #include <linux/errno.h>
>> @@ -549,15 +550,12 @@ void bus_probe_device(struct device *dev)
>>  {
>>         struct bus_type *bus = dev->bus;
>>         struct subsys_interface *sif;
>> -       int ret;
>>
>>         if (!bus)
>>                 return;
>>
>> -       if (bus->p->drivers_autoprobe) {
>> -               ret = device_attach(dev);
>> -               WARN_ON(ret < 0);
>> -       }
>> +       if (bus->p->drivers_autoprobe)
>> +               device_initial_probe(dev);
>>
>>         mutex_lock(&bus->p->mutex);
>>         list_for_each_entry(sif, &bus->p->interfaces, node)
>> @@ -659,6 +657,17 @@ static ssize_t uevent_store(struct device_driver *drv, const char *buf,
>>  }
>>  static DRIVER_ATTR_WO(uevent);
>>
>> +static void driver_attach_async(void *_drv, async_cookie_t cookie)
>> +{
>> +       struct device_driver *drv = _drv;
>> +       int ret;
>> +
>> +       ret = driver_attach(drv);
>> +
>> +       pr_debug("bus: '%s': driver %s async attach completed: %d\n",
>> +                drv->bus->name, drv->name, ret);
>> +}
>> +
>>  /**
>>   * bus_add_driver - Add a driver to the bus.
>>   * @drv: driver.
>> @@ -691,9 +700,15 @@ int bus_add_driver(struct device_driver *drv)
>>
>>         klist_add_tail(&priv->knode_bus, &bus->p->klist_drivers);
>>         if (drv->bus->p->drivers_autoprobe) {
>> -               error = driver_attach(drv);
>> -               if (error)
>> -                       goto out_unregister;
>> +               if (driver_allows_async_probing(drv)) {
>> +                       pr_debug("bus: '%s': probing driver %s asynchronously\n",
>> +                               drv->bus->name, drv->name);
>> +                       async_schedule(driver_attach_async, drv);
>> +               } else {
>> +                       error = driver_attach(drv);
>> +                       if (error)
>> +                               goto out_unregister;
>> +               }
>>         }
>>         module_add_driver(drv->owner, drv);
>>
>> diff --git a/drivers/base/dd.c b/drivers/base/dd.c
>> index e843fdb..2ad33b2 100644
>> --- a/drivers/base/dd.c
>> +++ b/drivers/base/dd.c
>> @@ -417,31 +417,95 @@ int driver_probe_device(struct device_driver *drv, struct device *dev)
>>         return ret;
>>  }
>>
>> -static int __device_attach(struct device_driver *drv, void *data)
>> +bool driver_allows_async_probing(struct device_driver *drv)
>>  {
>> -       struct device *dev = data;
>> +       return drv->probe_type == PROBE_PREFER_ASYNCHRONOUS;
>> +}
>> +
>> +struct device_attach_data {
>> +       struct device *dev;
>> +
>> +       /*
>> +        * Indicates whether we are are considering asynchronous probing or
>> +        * not. Only initial binding after device or driver registration
>> +        * (including deferral processing) may be done asynchronously, the
>> +        * rest is always synchronous, as we expect it is being done by
>> +        * request from userspace.
>> +        */
>> +       bool check_async;
>> +
>> +       /*
>> +        * Indicates if we are binding synchronous or asynchronous drivers.
>> +        * When asynchronous probing is enabled we'll execute 2 passes
>> +        * over drivers: first pass doing synchronous probing and second
>> +        * doing asynchronous probing (if synchronous did not succeed -
>> +        * most likely because there was no driver requiring synchronous
>> +        * probing - and we found asynchronous driver during first pass).
>> +        * The 2 passes are done because we can't shoot asynchronous
>> +        * probe for given device and driver from bus_for_each_drv() since
>> +        * driver pointer is not guaranteed to stay valid once
>> +        * bus_for_each_drv() iterates to the next driver on the bus.
>> +        */
>> +       bool want_async;
>> +
>> +       /*
>> +        * We'll set have_async to 'true' if, while scanning for matching
>> +        * driver, we'll encounter one that requests asynchronous probing.
>> +        */
>> +       bool have_async;
>> +};
>> +
>> +static int __device_attach_driver(struct device_driver *drv, void *_data)
>> +{
>> +       struct device_attach_data *data = _data;
>> +       struct device *dev = data->dev;
>> +       bool async_allowed;
>> +
>> +       /*
>> +        * Check if device has already been claimed. This may
>> +        * happen with driver loading, device discovery/registration,
>> +        * and deferred probe processing happens all at once with
>> +        * multiple threads.
>> +        */
>> +       if (dev->driver)
>> +               return -EBUSY;
>>
>>         if (!driver_match_device(drv, dev))
>>                 return 0;
>>
>> +       async_allowed = driver_allows_async_probing(drv);
>> +
>> +       if (async_allowed)
>> +               data->have_async = true;
>> +
>> +       if (data->check_async && async_allowed != data->want_async)
>> +               return 0;
>> +
>>         return driver_probe_device(drv, dev);
>>  }
>>
>> -/**
>> - * device_attach - try to attach device to a driver.
>> - * @dev: device.
>> - *
>> - * Walk the list of drivers that the bus has and call
>> - * driver_probe_device() for each pair. If a compatible
>> - * pair is found, break out and return.
>> - *
>> - * Returns 1 if the device was bound to a driver;
>> - * 0 if no matching driver was found;
>> - * -ENODEV if the device is not registered.
>> - *
>> - * When called for a USB interface, @dev->parent lock must be held.
>> - */
>> -int device_attach(struct device *dev)
>> +static void __device_attach_async_helper(void *_dev, async_cookie_t cookie)
>> +{
>> +       struct device *dev = _dev;
>> +       struct device_attach_data data = {
>> +               .dev            = dev,
>> +               .check_async    = true,
>> +               .want_async     = true,
>> +       };
>> +
>> +       device_lock(dev);
>> +
>> +       bus_for_each_drv(dev->bus, NULL, &data, __device_attach_driver);
>> +       dev_dbg(dev, "async probe completed\n");
>> +
>> +       pm_request_idle(dev);
>> +
>> +       device_unlock(dev);
>> +
>> +       put_device(dev);
>> +}
>> +
>> +int __device_attach(struct device *dev, bool allow_async)
>>  {
>>         int ret = 0;
>>
>> @@ -459,15 +523,59 @@ int device_attach(struct device *dev)
>>                         ret = 0;
>>                 }
>>         } else {
>> -               ret = bus_for_each_drv(dev->bus, NULL, dev, __device_attach);
>> -               pm_request_idle(dev);
>> +               struct device_attach_data data = {
>> +                       .dev = dev,
>> +                       .check_async = allow_async,
>> +                       .want_async = false,
>> +               };
>> +
>> +               ret = bus_for_each_drv(dev->bus, NULL, &data,
>> +                                       __device_attach_driver);
>> +               if (!ret && allow_async && data.have_async) {
>> +                       /*
>> +                        * If we could not find appropriate driver
>> +                        * synchronously and we are allowed to do
>> +                        * async probes and there are drivers that
>> +                        * want to probe asynchronously, we'll
>> +                        * try them.
>> +                        */
>> +                       dev_dbg(dev, "scheduling asynchronous probe\n");
>> +                       get_device(dev);
>> +                       async_schedule(__device_attach_async_helper, dev);
>> +               } else {
>> +                       pm_request_idle(dev);
>> +               }
>>         }
>>  out_unlock:
>>         device_unlock(dev);
>>         return ret;
>>  }
>> +
>> +/**
>> + * device_attach - try to attach device to a driver.
>> + * @dev: device.
>> + *
>> + * Walk the list of drivers that the bus has and call
>> + * driver_probe_device() for each pair. If a compatible
>> + * pair is found, break out and return.
>> + *
>> + * Returns 1 if the device was bound to a driver;
>> + * 0 if no matching driver was found;
>> + * -ENODEV if the device is not registered.
>> + *
>> + * When called for a USB interface, @dev->parent lock must be held.
>> + */
>> +int device_attach(struct device *dev)
>> +{
>> +       return __device_attach(dev, false);
>> +}
>>  EXPORT_SYMBOL_GPL(device_attach);
>>
>> +void device_initial_probe(struct device *dev)
>> +{
>> +       __device_attach(dev, true);
>> +}
>> +
>>  static int __driver_attach(struct device *dev, void *data)
>>  {
>>         struct device_driver *drv = data;
>> @@ -522,6 +630,9 @@ static void __device_release_driver(struct device *dev)
>>
>>         drv = dev->driver;
>>         if (drv) {
>> +               if (driver_allows_async_probing(drv))
>> +                       async_synchronize_full();
>> +
>>                 pm_runtime_get_sync(dev);
>>
>>                 driver_sysfs_remove(dev);
>> diff --git a/include/linux/device.h b/include/linux/device.h
>> index 884aa6e..400cacd 100644
>> --- a/include/linux/device.h
>> +++ b/include/linux/device.h
>> @@ -196,12 +196,38 @@ extern struct kset *bus_get_kset(struct bus_type *bus);
>>  extern struct klist *bus_get_device_klist(struct bus_type *bus);
>>
>>  /**
>> + * enum probe_type - device driver probe type to try
>> + *     Device drivers may opt in for special handling of their
>> + *     respective probe routines. This tells the core what to
>> + *     expect and prefer.
>> + *
>> + * @PROBE_SYNCHRONOUS: Default. Drivers expect their probe routines
>> + *     to run synchronously with driver and device registration
>> + *     (with the exception of -EPROBE_DEFER handling - re-probing
>> + *     always ends up being done asynchronously).
>> + * @PROBE_PREFER_ASYNCHRONOUS: Drivers for "slow" devices which
>> + *     probing order is not essential for booting the system may
>> + *     opt into executing their probes asynchronously.
>> + *
>> + * Note that the end goal is to switch the kernel to use asynchronous
>> + * probing by default, so annotating drivers with
>> + * %PROBE_PREFER_ASYNCHRONOUS is a temporary measure that allows us
>> + * to speed up boot process while we are validating the rest of the
>> + * drivers.
>> + */
>> +enum probe_type {
>> +       PROBE_SYNCHRONOUS,
>> +       PROBE_PREFER_ASYNCHRONOUS,
>> +};
>> +
>> +/**
>>   * struct device_driver - The basic device driver structure
>>   * @name:      Name of the device driver.
>>   * @bus:       The bus which the device of this driver belongs to.
>>   * @owner:     The module owner.
>>   * @mod_name:  Used for built-in modules.
>>   * @suppress_bind_attrs: Disables bind/unbind via sysfs.
>> + * @probe_type:        Type of the probe (synchronous or asynchronous) to use.
>>   * @of_match_table: The open firmware table.
>>   * @acpi_match_table: The ACPI match table.
>>   * @probe:     Called to query the existence of a specific device,
>> @@ -235,6 +261,7 @@ struct device_driver {
>>         const char              *mod_name;      /* used for built-in modules */
>>
>>         bool suppress_bind_attrs;       /* disables bind/unbind via sysfs */
>> +       enum probe_type probe_type;
>>
>>         const struct of_device_id       *of_match_table;
>>         const struct acpi_device_id     *acpi_match_table;
>> @@ -972,6 +999,7 @@ extern int __must_check device_bind_driver(struct device *dev);
>>  extern void device_release_driver(struct device *dev);
>>  extern int  __must_check device_attach(struct device *dev);
>>  extern int __must_check driver_attach(struct device_driver *drv);
>> +extern void device_initial_probe(struct device *dev);
>>  extern int __must_check device_reprobe(struct device *dev);
>>
>>  /*
>> --
>> 2.2.0.rc0.207.ga3a616c
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Tomeu Vizoso]

On 29 May 2015 at 15:23, Tomeu Vizoso <tomeu.vizoso@gmail.com> wrote:
> On 29 May 2015 at 12:48, Tomeu Vizoso <tomeu.vizoso@gmail.com> wrote:
>> On 31 March 2015 at 01:20, Dmitry Torokhov <dmitry.torokhov@gmail.com> wrote:
>>> Some devices take a long time when initializing, and not all drivers are
>>> suited to initialize their devices when they are open. For example,
>>> input drivers need to interrogate their devices in order to publish
>>> device's capabilities before userspace will open them. When such drivers
>>> are compiled into kernel they may stall entire kernel initialization.
>>>
>>> This change allows drivers request for their probe functions to be
>>> called asynchronously during driver and device registration (manual
>>> binding is still synchronous). Because async_schedule is used to perform
>>> asynchronous calls module loading will still wait for the probing to
>>> complete.
>>
>> But what about parents? Don't we need to make sure that before probing
>> a device its parent has finished probing already?
>
> Have realized that this is an existing problem that was just made more
> probable by async probe, as before async probing the parent could have
> deferred its probe and then its children would be probed.
>
> Wonder if drivers should be modified to defer its probe if their
> parent isn't probed yet, or if we can codify that in dd.c.

Also wonder what's the plan regarding USB interfaces requiring that
their parent's lock is taken before probing.

Thanks,

Tomeu

> Regards,
>
> Tomeu
>
>> This backtrace
>> illustrates the problem:
>>
>> [<c0014818>] (__dabt_svc) from [<c03737ac>] (host1x_syncpt_alloc+0x14/0x134)
>> [<c03737ac>] (host1x_syncpt_alloc) from [<c03738f4>]
>> (host1x_syncpt_request+0x28/0x2c)
>> [<c03738f4>] (host1x_syncpt_request) from [<c03b55ec>]
>> (tegra_dc_probe+0x198/0x35c)
>> [<c03b55ec>] (tegra_dc_probe) from [<c03cb5a0>] (platform_drv_probe+0x54/0xbc)
>> [<c03cb5a0>] (platform_drv_probe) from [<c03c96e0>]
>> (driver_probe_device+0x184/0x2c4)
>> [<c03c96e0>] (driver_probe_device) from [<c03c98bc>] (__driver_attach+0x9c/0xa0)
>> [<c03c98bc>] (__driver_attach) from [<c03c78d8>] (bus_for_each_dev+0x78/0xac)
>> [<c03c78d8>] (bus_for_each_dev) from [<c03c9070>] (driver_attach+0x2c/0x30)
>> [<c03c9070>] (driver_attach) from [<c03c7e10>] (driver_attach_async+0x18/0x1c)
>> [<c03c7e10>] (driver_attach_async) from [<c004afd0>]
>> (async_run_entry_fn+0x58/0x128)
>> [<c004afd0>] (async_run_entry_fn) from [<c0042470>]
>> (process_one_work+0x140/0x50c)
>> [<c0042470>] (process_one_work) from [<c0042890>] (worker_thread+0x54/0x52c)
>> [<c0042890>] (worker_thread) from [<c0048554>] (kthread+0xec/0x104)
>> [<c0048554>] (kthread) from [<c000fc28>] (ret_from_fork+0x14/0x2c)
>>
>> host1x_syncpt_request() assumes that the parent of the DC (host1x) has
>> been probed already and its drvdata is available.
>>
>> Thanks,
>>
>> Tomeu
>>
>>> Note that the end goal is to make the probing asynchronous by default,
>>> so annotating drivers with PROBE_PREFER_ASYNCHRONOUS is a temporary
>>> measure that allows us to speed up boot process while we validating and
>>> fixing the rest of the drivers and preparing userspace.
>>>
>>> This change is based on earlier patch by "Luis R. Rodriguez"
>>> <mcgrof@suse.com>
>>>
>>> Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
>>> ---
>>>  drivers/base/base.h    |   1 +
>>>  drivers/base/bus.c     |  31 +++++++---
>>>  drivers/base/dd.c      | 149 ++++++++++++++++++++++++++++++++++++++++++-------
>>>  include/linux/device.h |  28 ++++++++++
>>>  4 files changed, 182 insertions(+), 27 deletions(-)
>>>
>>> diff --git a/drivers/base/base.h b/drivers/base/base.h
>>> index 251c5d3..fd3347d 100644
>>> --- a/drivers/base/base.h
>>> +++ b/drivers/base/base.h
>>> @@ -116,6 +116,7 @@ static inline int driver_match_device(struct device_driver *drv,
>>>  {
>>>         return drv->bus->match ? drv->bus->match(dev, drv) : 1;
>>>  }
>>> +extern bool driver_allows_async_probing(struct device_driver *drv);
>>>
>>>  extern int driver_add_groups(struct device_driver *drv,
>>>                              const struct attribute_group **groups);
>>> diff --git a/drivers/base/bus.c b/drivers/base/bus.c
>>> index 79bc203..5005924 100644
>>> --- a/drivers/base/bus.c
>>> +++ b/drivers/base/bus.c
>>> @@ -10,6 +10,7 @@
>>>   *
>>>   */
>>>
>>> +#include <linux/async.h>
>>>  #include <linux/device.h>
>>>  #include <linux/module.h>
>>>  #include <linux/errno.h>
>>> @@ -549,15 +550,12 @@ void bus_probe_device(struct device *dev)
>>>  {
>>>         struct bus_type *bus = dev->bus;
>>>         struct subsys_interface *sif;
>>> -       int ret;
>>>
>>>         if (!bus)
>>>                 return;
>>>
>>> -       if (bus->p->drivers_autoprobe) {
>>> -               ret = device_attach(dev);
>>> -               WARN_ON(ret < 0);
>>> -       }
>>> +       if (bus->p->drivers_autoprobe)
>>> +               device_initial_probe(dev);
>>>
>>>         mutex_lock(&bus->p->mutex);
>>>         list_for_each_entry(sif, &bus->p->interfaces, node)
>>> @@ -659,6 +657,17 @@ static ssize_t uevent_store(struct device_driver *drv, const char *buf,
>>>  }
>>>  static DRIVER_ATTR_WO(uevent);
>>>
>>> +static void driver_attach_async(void *_drv, async_cookie_t cookie)
>>> +{
>>> +       struct device_driver *drv = _drv;
>>> +       int ret;
>>> +
>>> +       ret = driver_attach(drv);
>>> +
>>> +       pr_debug("bus: '%s': driver %s async attach completed: %d\n",
>>> +                drv->bus->name, drv->name, ret);
>>> +}
>>> +
>>>  /**
>>>   * bus_add_driver - Add a driver to the bus.
>>>   * @drv: driver.
>>> @@ -691,9 +700,15 @@ int bus_add_driver(struct device_driver *drv)
>>>
>>>         klist_add_tail(&priv->knode_bus, &bus->p->klist_drivers);
>>>         if (drv->bus->p->drivers_autoprobe) {
>>> -               error = driver_attach(drv);
>>> -               if (error)
>>> -                       goto out_unregister;
>>> +               if (driver_allows_async_probing(drv)) {
>>> +                       pr_debug("bus: '%s': probing driver %s asynchronously\n",
>>> +                               drv->bus->name, drv->name);
>>> +                       async_schedule(driver_attach_async, drv);
>>> +               } else {
>>> +                       error = driver_attach(drv);
>>> +                       if (error)
>>> +                               goto out_unregister;
>>> +               }
>>>         }
>>>         module_add_driver(drv->owner, drv);
>>>
>>> diff --git a/drivers/base/dd.c b/drivers/base/dd.c
>>> index e843fdb..2ad33b2 100644
>>> --- a/drivers/base/dd.c
>>> +++ b/drivers/base/dd.c
>>> @@ -417,31 +417,95 @@ int driver_probe_device(struct device_driver *drv, struct device *dev)
>>>         return ret;
>>>  }
>>>
>>> -static int __device_attach(struct device_driver *drv, void *data)
>>> +bool driver_allows_async_probing(struct device_driver *drv)
>>>  {
>>> -       struct device *dev = data;
>>> +       return drv->probe_type == PROBE_PREFER_ASYNCHRONOUS;
>>> +}
>>> +
>>> +struct device_attach_data {
>>> +       struct device *dev;
>>> +
>>> +       /*
>>> +        * Indicates whether we are are considering asynchronous probing or
>>> +        * not. Only initial binding after device or driver registration
>>> +        * (including deferral processing) may be done asynchronously, the
>>> +        * rest is always synchronous, as we expect it is being done by
>>> +        * request from userspace.
>>> +        */
>>> +       bool check_async;
>>> +
>>> +       /*
>>> +        * Indicates if we are binding synchronous or asynchronous drivers.
>>> +        * When asynchronous probing is enabled we'll execute 2 passes
>>> +        * over drivers: first pass doing synchronous probing and second
>>> +        * doing asynchronous probing (if synchronous did not succeed -
>>> +        * most likely because there was no driver requiring synchronous
>>> +        * probing - and we found asynchronous driver during first pass).
>>> +        * The 2 passes are done because we can't shoot asynchronous
>>> +        * probe for given device and driver from bus_for_each_drv() since
>>> +        * driver pointer is not guaranteed to stay valid once
>>> +        * bus_for_each_drv() iterates to the next driver on the bus.
>>> +        */
>>> +       bool want_async;
>>> +
>>> +       /*
>>> +        * We'll set have_async to 'true' if, while scanning for matching
>>> +        * driver, we'll encounter one that requests asynchronous probing.
>>> +        */
>>> +       bool have_async;
>>> +};
>>> +
>>> +static int __device_attach_driver(struct device_driver *drv, void *_data)
>>> +{
>>> +       struct device_attach_data *data = _data;
>>> +       struct device *dev = data->dev;
>>> +       bool async_allowed;
>>> +
>>> +       /*
>>> +        * Check if device has already been claimed. This may
>>> +        * happen with driver loading, device discovery/registration,
>>> +        * and deferred probe processing happens all at once with
>>> +        * multiple threads.
>>> +        */
>>> +       if (dev->driver)
>>> +               return -EBUSY;
>>>
>>>         if (!driver_match_device(drv, dev))
>>>                 return 0;
>>>
>>> +       async_allowed = driver_allows_async_probing(drv);
>>> +
>>> +       if (async_allowed)
>>> +               data->have_async = true;
>>> +
>>> +       if (data->check_async && async_allowed != data->want_async)
>>> +               return 0;
>>> +
>>>         return driver_probe_device(drv, dev);
>>>  }
>>>
>>> -/**
>>> - * device_attach - try to attach device to a driver.
>>> - * @dev: device.
>>> - *
>>> - * Walk the list of drivers that the bus has and call
>>> - * driver_probe_device() for each pair. If a compatible
>>> - * pair is found, break out and return.
>>> - *
>>> - * Returns 1 if the device was bound to a driver;
>>> - * 0 if no matching driver was found;
>>> - * -ENODEV if the device is not registered.
>>> - *
>>> - * When called for a USB interface, @dev->parent lock must be held.
>>> - */
>>> -int device_attach(struct device *dev)
>>> +static void __device_attach_async_helper(void *_dev, async_cookie_t cookie)
>>> +{
>>> +       struct device *dev = _dev;
>>> +       struct device_attach_data data = {
>>> +               .dev            = dev,
>>> +               .check_async    = true,
>>> +               .want_async     = true,
>>> +       };
>>> +
>>> +       device_lock(dev);
>>> +
>>> +       bus_for_each_drv(dev->bus, NULL, &data, __device_attach_driver);
>>> +       dev_dbg(dev, "async probe completed\n");
>>> +
>>> +       pm_request_idle(dev);
>>> +
>>> +       device_unlock(dev);
>>> +
>>> +       put_device(dev);
>>> +}
>>> +
>>> +int __device_attach(struct device *dev, bool allow_async)
>>>  {
>>>         int ret = 0;
>>>
>>> @@ -459,15 +523,59 @@ int device_attach(struct device *dev)
>>>                         ret = 0;
>>>                 }
>>>         } else {
>>> -               ret = bus_for_each_drv(dev->bus, NULL, dev, __device_attach);
>>> -               pm_request_idle(dev);
>>> +               struct device_attach_data data = {
>>> +                       .dev = dev,
>>> +                       .check_async = allow_async,
>>> +                       .want_async = false,
>>> +               };
>>> +
>>> +               ret = bus_for_each_drv(dev->bus, NULL, &data,
>>> +                                       __device_attach_driver);
>>> +               if (!ret && allow_async && data.have_async) {
>>> +                       /*
>>> +                        * If we could not find appropriate driver
>>> +                        * synchronously and we are allowed to do
>>> +                        * async probes and there are drivers that
>>> +                        * want to probe asynchronously, we'll
>>> +                        * try them.
>>> +                        */
>>> +                       dev_dbg(dev, "scheduling asynchronous probe\n");
>>> +                       get_device(dev);
>>> +                       async_schedule(__device_attach_async_helper, dev);
>>> +               } else {
>>> +                       pm_request_idle(dev);
>>> +               }
>>>         }
>>>  out_unlock:
>>>         device_unlock(dev);
>>>         return ret;
>>>  }
>>> +
>>> +/**
>>> + * device_attach - try to attach device to a driver.
>>> + * @dev: device.
>>> + *
>>> + * Walk the list of drivers that the bus has and call
>>> + * driver_probe_device() for each pair. If a compatible
>>> + * pair is found, break out and return.
>>> + *
>>> + * Returns 1 if the device was bound to a driver;
>>> + * 0 if no matching driver was found;
>>> + * -ENODEV if the device is not registered.
>>> + *
>>> + * When called for a USB interface, @dev->parent lock must be held.
>>> + */
>>> +int device_attach(struct device *dev)
>>> +{
>>> +       return __device_attach(dev, false);
>>> +}
>>>  EXPORT_SYMBOL_GPL(device_attach);
>>>
>>> +void device_initial_probe(struct device *dev)
>>> +{
>>> +       __device_attach(dev, true);
>>> +}
>>> +
>>>  static int __driver_attach(struct device *dev, void *data)
>>>  {
>>>         struct device_driver *drv = data;
>>> @@ -522,6 +630,9 @@ static void __device_release_driver(struct device *dev)
>>>
>>>         drv = dev->driver;
>>>         if (drv) {
>>> +               if (driver_allows_async_probing(drv))
>>> +                       async_synchronize_full();
>>> +
>>>                 pm_runtime_get_sync(dev);
>>>
>>>                 driver_sysfs_remove(dev);
>>> diff --git a/include/linux/device.h b/include/linux/device.h
>>> index 884aa6e..400cacd 100644
>>> --- a/include/linux/device.h
>>> +++ b/include/linux/device.h
>>> @@ -196,12 +196,38 @@ extern struct kset *bus_get_kset(struct bus_type *bus);
>>>  extern struct klist *bus_get_device_klist(struct bus_type *bus);
>>>
>>>  /**
>>> + * enum probe_type - device driver probe type to try
>>> + *     Device drivers may opt in for special handling of their
>>> + *     respective probe routines. This tells the core what to
>>> + *     expect and prefer.
>>> + *
>>> + * @PROBE_SYNCHRONOUS: Default. Drivers expect their probe routines
>>> + *     to run synchronously with driver and device registration
>>> + *     (with the exception of -EPROBE_DEFER handling - re-probing
>>> + *     always ends up being done asynchronously).
>>> + * @PROBE_PREFER_ASYNCHRONOUS: Drivers for "slow" devices which
>>> + *     probing order is not essential for booting the system may
>>> + *     opt into executing their probes asynchronously.
>>> + *
>>> + * Note that the end goal is to switch the kernel to use asynchronous
>>> + * probing by default, so annotating drivers with
>>> + * %PROBE_PREFER_ASYNCHRONOUS is a temporary measure that allows us
>>> + * to speed up boot process while we are validating the rest of the
>>> + * drivers.
>>> + */
>>> +enum probe_type {
>>> +       PROBE_SYNCHRONOUS,
>>> +       PROBE_PREFER_ASYNCHRONOUS,
>>> +};
>>> +
>>> +/**
>>>   * struct device_driver - The basic device driver structure
>>>   * @name:      Name of the device driver.
>>>   * @bus:       The bus which the device of this driver belongs to.
>>>   * @owner:     The module owner.
>>>   * @mod_name:  Used for built-in modules.
>>>   * @suppress_bind_attrs: Disables bind/unbind via sysfs.
>>> + * @probe_type:        Type of the probe (synchronous or asynchronous) to use.
>>>   * @of_match_table: The open firmware table.
>>>   * @acpi_match_table: The ACPI match table.
>>>   * @probe:     Called to query the existence of a specific device,
>>> @@ -235,6 +261,7 @@ struct device_driver {
>>>         const char              *mod_name;      /* used for built-in modules */
>>>
>>>         bool suppress_bind_attrs;       /* disables bind/unbind via sysfs */
>>> +       enum probe_type probe_type;
>>>
>>>         const struct of_device_id       *of_match_table;
>>>         const struct acpi_device_id     *acpi_match_table;
>>> @@ -972,6 +999,7 @@ extern int __must_check device_bind_driver(struct device *dev);
>>>  extern void device_release_driver(struct device *dev);
>>>  extern int  __must_check device_attach(struct device *dev);
>>>  extern int __must_check driver_attach(struct device_driver *drv);
>>> +extern void device_initial_probe(struct device *dev);
>>>  extern int __must_check device_reprobe(struct device *dev);
>>>
>>>  /*
>>> --
>>> 2.2.0.rc0.207.ga3a616c
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Dmitry Torokhov]

On Mon, Jun 01, 2015 at 02:04:01PM +0200, Tomeu Vizoso wrote:
> On 29 May 2015 at 15:23, Tomeu Vizoso <tomeu.vizoso@gmail.com> wrote:
> > On 29 May 2015 at 12:48, Tomeu Vizoso <tomeu.vizoso@gmail.com> wrote:
> >> On 31 March 2015 at 01:20, Dmitry Torokhov <dmitry.torokhov@gmail.com> wrote:
> >>> Some devices take a long time when initializing, and not all drivers are
> >>> suited to initialize their devices when they are open. For example,
> >>> input drivers need to interrogate their devices in order to publish
> >>> device's capabilities before userspace will open them. When such drivers
> >>> are compiled into kernel they may stall entire kernel initialization.
> >>>
> >>> This change allows drivers request for their probe functions to be
> >>> called asynchronously during driver and device registration (manual
> >>> binding is still synchronous). Because async_schedule is used to perform
> >>> asynchronous calls module loading will still wait for the probing to
> >>> complete.
> >>
> >> But what about parents? Don't we need to make sure that before probing
> >> a device its parent has finished probing already?
> >
> > Have realized that this is an existing problem that was just made more
> > probable by async probe, as before async probing the parent could have
> > deferred its probe and then its children would be probed.
> >
> > Wonder if drivers should be modified to defer its probe if their
> > parent isn't probed yet, or if we can codify that in dd.c.
> 
> Also wonder what's the plan regarding USB interfaces requiring that
> their parent's lock is taken before probing.

Yes, indeed, we need to take paren's lock in async probe too. I'll make
a patch.

Thanks for spotting this.

-- 
Dmitry

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Dan Williams]

On Mon, Mar 30, 2015 at 4:20 PM, Dmitry Torokhov
<dmitry.torokhov@gmail.com> wrote:
> Some devices take a long time when initializing, and not all drivers are
> suited to initialize their devices when they are open. For example,
> input drivers need to interrogate their devices in order to publish
> device's capabilities before userspace will open them. When such drivers
> are compiled into kernel they may stall entire kernel initialization.
>
> This change allows drivers request for their probe functions to be
> called asynchronously during driver and device registration (manual
> binding is still synchronous). Because async_schedule is used to perform
> asynchronous calls module loading will still wait for the probing to
> complete.
>
> Note that the end goal is to make the probing asynchronous by default,
> so annotating drivers with PROBE_PREFER_ASYNCHRONOUS is a temporary
> measure that allows us to speed up boot process while we validating and
> fixing the rest of the drivers and preparing userspace.
>
> This change is based on earlier patch by "Luis R. Rodriguez"
> <mcgrof@suse.com>
>
> Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
> ---
>  drivers/base/base.h    |   1 +
>  drivers/base/bus.c     |  31 +++++++---
>  drivers/base/dd.c      | 149 ++++++++++++++++++++++++++++++++++++++++++-------
>  include/linux/device.h |  28 ++++++++++
>  4 files changed, 182 insertions(+), 27 deletions(-)

Just noticed this patch.  It caught my eye because I had a hard time
getting an open coded implementation of asynchronous probing to work
in the new libnvdimm subsystem.  Especially the messy races of tearing
things down while probing is still in flight.  I ended up implementing
asynchronous device registration which eliminated a lot of complexity
and of course the bugs.  In general I tend to think that async
registration is less risky than async probe since it keeps wider
portions of the traditional device model synchronous and leverages the
fact that the device model is already well prepared for asynchronous
arrival of devices due to hotplug.  Splitting the "initial probe" from
the "manual probe" case seems like a recipe for confusion.

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Luis R. Rodriguez]

On Sat, Jun 27, 2015 at 04:45:25PM -0700, Dan Williams wrote:
> On Mon, Mar 30, 2015 at 4:20 PM, Dmitry Torokhov
> <dmitry.torokhov@gmail.com> wrote:
> > Some devices take a long time when initializing, and not all drivers are
> > suited to initialize their devices when they are open. For example,
> > input drivers need to interrogate their devices in order to publish
> > device's capabilities before userspace will open them. When such drivers
> > are compiled into kernel they may stall entire kernel initialization.
> >
> > This change allows drivers request for their probe functions to be
> > called asynchronously during driver and device registration (manual
> > binding is still synchronous). Because async_schedule is used to perform
> > asynchronous calls module loading will still wait for the probing to
> > complete.
> >
> > Note that the end goal is to make the probing asynchronous by default,
> > so annotating drivers with PROBE_PREFER_ASYNCHRONOUS is a temporary
> > measure that allows us to speed up boot process while we validating and
> > fixing the rest of the drivers and preparing userspace.
> >
> > This change is based on earlier patch by "Luis R. Rodriguez"
> > <mcgrof@suse.com>
> >
> > Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
> > ---
> >  drivers/base/base.h    |   1 +
> >  drivers/base/bus.c     |  31 +++++++---
> >  drivers/base/dd.c      | 149 ++++++++++++++++++++++++++++++++++++++++++-------
> >  include/linux/device.h |  28 ++++++++++
> >  4 files changed, 182 insertions(+), 27 deletions(-)
> 
> Just noticed this patch.  It caught my eye because I had a hard time
> getting an open coded implementation of asynchronous probing to work
> in the new libnvdimm subsystem.  Especially the messy races of tearing
> things down while probing is still in flight.  I ended up implementing
> asynchronous device registration which eliminated a lot of complexity
> and of course the bugs.  In general I tend to think that async
> registration is less risky than async probe since it keeps wider
> portions of the traditional device model synchronous 

but its not see -DEFER_PROBE even before async probe.

> and leverages the
> fact that the device model is already well prepared for asynchronous
> arrival of devices due to hotplug.

I think this sounds reasonable, do you have your code upstream or posted?
If not will you be at Plumbers? Maybe we shoudl talk about this as although
ChromeOS already likely already jumped on async probe we should address a
way forward and path forward for other distributions and I don't think anyone
is looking too much into it. async probe came to Linux for two reasons:

 * chromeos wanting it
 * an incorrect systemd assumption on how the driver core works

So long term we still need to address the systemd approach, are they going
to be defaulting now to async probe for all modules? How about for built-ins?

We should talk about this and maybe at plumbers.

> Splitting the "initial probe" from
> the "manual probe" case seems like a recipe for confusion.

If you can come up with pros / cons on both strategies it'd be
valuable.

  Luis

Re: [PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Dmitry Torokhov]

On Sat, Jun 27, 2015 at 04:45:25PM -0700, Dan Williams wrote:
> On Mon, Mar 30, 2015 at 4:20 PM, Dmitry Torokhov
> <dmitry.torokhov@gmail.com> wrote:
> > Some devices take a long time when initializing, and not all drivers are
> > suited to initialize their devices when they are open. For example,
> > input drivers need to interrogate their devices in order to publish
> > device's capabilities before userspace will open them. When such drivers
> > are compiled into kernel they may stall entire kernel initialization.
> >
> > This change allows drivers request for their probe functions to be
> > called asynchronously during driver and device registration (manual
> > binding is still synchronous). Because async_schedule is used to perform
> > asynchronous calls module loading will still wait for the probing to
> > complete.
> >
> > Note that the end goal is to make the probing asynchronous by default,
> > so annotating drivers with PROBE_PREFER_ASYNCHRONOUS is a temporary
> > measure that allows us to speed up boot process while we validating and
> > fixing the rest of the drivers and preparing userspace.
> >
> > This change is based on earlier patch by "Luis R. Rodriguez"
> > <mcgrof@suse.com>
> >
> > Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
> > ---
> >  drivers/base/base.h    |   1 +
> >  drivers/base/bus.c     |  31 +++++++---
> >  drivers/base/dd.c      | 149 ++++++++++++++++++++++++++++++++++++++++++-------
> >  include/linux/device.h |  28 ++++++++++
> >  4 files changed, 182 insertions(+), 27 deletions(-)
> 
> Just noticed this patch.  It caught my eye because I had a hard time
> getting an open coded implementation of asynchronous probing to work
> in the new libnvdimm subsystem.  Especially the messy races of tearing
> things down while probing is still in flight.

And that is exactly the reason why asynchronous probing was moved into
driver code. It knows the state of the device and knows when it is OK to
remove it or start suspend transitions, etc.

> I ended up implementing
> asynchronous device registration which eliminated a lot of complexity
> and of course the bugs.

serio and gameport subsystems have been using asynchronous registration
for ages (not because of probing but because of issues with serio ports
- such as Synaptics pass-through - stacked on top of each other and
historicaly driver code deadlocking if you try to register child
device on the same bus that parent is). However I was never too happy
with it because with asynchronous registration you can't really handle
errors. What do you do if device registartion fails?

> In general I tend to think that async
> registration is less risky than async probe since it keeps wider
> portions of the traditional device model synchronous and leverages the
> fact that the device model is already well prepared for asynchronous
> arrival of devices due to hotplug.  Splitting the "initial probe" from
> the "manual probe" case seems like a recipe for confusion.

The split is because again serio and USB subsystems resort to somewhat
manual binding due to the need of handling recursion on the bus. But
otherwise we already have asynchronous probing, because we have deferred
probes and also driver modules can be loaded asynchronously with device
registration.

If you have concrete examples where asynchronous probig as it merged
causes issues I'd love to hear and fix them.

Thanks.

-- 
Dmitry

[PATCH 0/8] Asynchronous device/driver probing support

[Dmitry Torokhov]

This series is a combination of changes proposed by Luis a couple months
ago and implementation used by Chrome OS. The issue we are trying to solve
here is "slow" devices and drivers spending "too much time" in their probe()
methods and it affects:

- overall kernel boot process when drivers are compiled into the kernel
  and slow devices stall entire boot progress;
- systemd desire to time out module loading process.

Unlike Luis' proposal we do make use of asycn_schedule() infrastructure
instead of using a dedicated workqueue, so all  existing synchronization
points in kernel that wait for device registration still work the same.
Also, the asynchronous probing is done not only during driver registration
(i.e. when devices are probed asynchronously only if they are registered
before the driver), but also during device registration and deferred probe
handling. This way slow devices do not stall kernel boot even when drivers
are compiled into the kernel.

The last patch is for adventurous people to try and force
fully-asynchronous boot. It works for me with limited success - I can boot
Rockhip-based box to userspace as long as I force serial to be sychronously
probed and ignore the fact that most devices are using "dummy" regulators
as regulator subsystem really expects regulators to be registered in
orderly fashion on OF-based systems.

Thanks,
Dmitry


Dmitry Torokhov (3):
  driver-core: add asynchronous probing support for drivers
  driver-core: platform_driver_probe() must probe synchronously
  module: add core_param_unsafe

Luis R. Rodriguez (5):
  module: add extra argument for parse_params() callback
  driver-core: add driver module asynchronous probe support
  driver-core: enable drivers to opt-out of async probe
  amd64_edac: enforce synchronous probe
  driver-core: allow forcing async probing for modules and builtins

 Documentation/kernel-parameters.txt |  13 +++
 arch/powerpc/mm/hugetlbpage.c       |   4 +-
 drivers/base/base.h                 |   1 +
 drivers/base/bus.c                  |  31 +++++--
 drivers/base/dd.c                   | 166 +++++++++++++++++++++++++++++++-----
 drivers/base/platform.c             |  13 +++
 drivers/edac/amd64_edac.c           |   1 +
 include/linux/device.h              |  26 ++++++
 include/linux/module.h              |   2 +
 include/linux/moduleparam.h         |  12 ++-
 init/main.c                         |  25 +++---
 kernel/module.c                     |  25 +++++-
 kernel/params.c                     |  11 ++-
 lib/dynamic_debug.c                 |   4 +-
 14 files changed, 284 insertions(+), 50 deletions(-)

-- 
2.2.0.rc0.207.ga3a616c

[PATCH 2/8] driver-core: add asynchronous probing support for drivers

[Dmitry Torokhov]

Some devices take a long time when initializing, and not all drivers are
suited to initialize their devices when they are open. For example,
input drivers need to interrogate their devices in order to publish
device's capabilities before userspace will open them. When such drivers
are compiled into kernel they may stall entire kernel initialization.

This change allows drivers request for their probe functions to be
called asynchronously during driver and device registration (manual
binding is still synchronous). Because async_schedule is used to perform
asynchronous calls module loading will still wait for the probing to
complete.

This change is based on earlier patch by "Luis R. Rodriguez"
<mcgrof@suse.com>

Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
---
 drivers/base/base.h    |   1 +
 drivers/base/bus.c     |  31 +++++++---
 drivers/base/dd.c      | 149 ++++++++++++++++++++++++++++++++++++++++++-------
 include/linux/device.h |  21 +++++++
 4 files changed, 175 insertions(+), 27 deletions(-)

diff --git a/drivers/base/base.h b/drivers/base/base.h
index 251c5d3..fd3347d 100644
--- a/drivers/base/base.h
+++ b/drivers/base/base.h
@@ -116,6 +116,7 @@ static inline int driver_match_device(struct device_driver *drv,
 {
 	return drv->bus->match ? drv->bus->match(dev, drv) : 1;
 }
+extern bool driver_allows_async_probing(struct device_driver *drv);
 
 extern int driver_add_groups(struct device_driver *drv,
 			     const struct attribute_group **groups);
diff --git a/drivers/base/bus.c b/drivers/base/bus.c
index 876bae5..a3d71ad 100644
--- a/drivers/base/bus.c
+++ b/drivers/base/bus.c
@@ -10,6 +10,7 @@
  *
  */
 
+#include <linux/async.h>
 #include <linux/device.h>
 #include <linux/module.h>
 #include <linux/errno.h>
@@ -549,15 +550,12 @@ void bus_probe_device(struct device *dev)
 {
 	struct bus_type *bus = dev->bus;
 	struct subsys_interface *sif;
-	int ret;
 
 	if (!bus)
 		return;
 
-	if (bus->p->drivers_autoprobe) {
-		ret = device_attach(dev);
-		WARN_ON(ret < 0);
-	}
+	if (bus->p->drivers_autoprobe)
+		device_initial_probe(dev);
 
 	mutex_lock(&bus->p->mutex);
 	list_for_each_entry(sif, &bus->p->interfaces, node)
@@ -659,6 +657,17 @@ static ssize_t uevent_store(struct device_driver *drv, const char *buf,
 }
 static DRIVER_ATTR_WO(uevent);
 
+static void driver_attach_async(void *_drv, async_cookie_t cookie)
+{
+	struct device_driver *drv = _drv;
+	int ret;
+
+	ret = driver_attach(drv);
+
+	pr_debug("bus: '%s': driver %s async attach completed: %d\n",
+		 drv->bus->name, drv->name, ret);
+}
+
 /**
  * bus_add_driver - Add a driver to the bus.
  * @drv: driver.
@@ -691,9 +700,15 @@ int bus_add_driver(struct device_driver *drv)
 
 	klist_add_tail(&priv->knode_bus, &bus->p->klist_drivers);
 	if (drv->bus->p->drivers_autoprobe) {
-		error = driver_attach(drv);
-		if (error)
-			goto out_unregister;
+		if (driver_allows_async_probing(drv)) {
+			pr_debug("bus: '%s': probing driver %s asynchronously\n",
+				drv->bus->name, drv->name);
+			async_schedule(driver_attach_async, drv);
+		} else {
+			error = driver_attach(drv);
+			if (error)
+				goto out_unregister;
+		}
 	}
 	module_add_driver(drv->owner, drv);
 
diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index cdc779c..f9f4831 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -402,31 +402,95 @@ int driver_probe_device(struct device_driver *drv, struct device *dev)
 	return ret;
 }
 
-static int __device_attach(struct device_driver *drv, void *data)
+bool driver_allows_async_probing(struct device_driver *drv)
 {
-	struct device *dev = data;
+	return drv->probe_type == PROBE_PREFER_ASYNCHRONOUS;
+}
+
+struct device_attach_data {
+	struct device *dev;
+
+	/*
+	 * Indicates whether we are are considering asynchronous probing or
+	 * not. Only initial binding after device or driver registration
+	 * (including deferral processing) may be done asynchronously, the
+	 * rest is always synchronous, as we expect it is being done by
+	 * request from userspace.
+	 */
+	bool check_async;
+
+	/*
+	 * Indicates if we are binding synchronous or asynchronous drivers.
+	 * When asynchronous probing is enabled we'll execute 2 passes
+	 * over drivers: first pass doing synchronous probing and second
+	 * doing asynchronous probing (if synchronous did not succeed -
+	 * most likely because there was no driver requiring synchronous
+	 * probing - and we found asynchronous driver during first pass).
+	 * The 2 passes are done because we can't shoot asynchronous
+	 * probe for given device and driver from bus_for_each_drv() since
+	 * driver pointer is not guaranteed to stay valid once
+	 * bus_for_each_drv() iterates to the next driver on the bus.
+	 */
+	bool want_async;
+
+	/*
+	 * We'll set have_async to 'true' if, while scanning for matching
+	 * driver, we'll encounter one that requests asynchronous probing.
+	 */
+	bool have_async;
+};
+
+static int __device_attach_driver(struct device_driver *drv, void *_data)
+{
+	struct device_attach_data *data = _data;
+	struct device *dev = data->dev;
+	bool async_allowed;
+
+	/*
+	 * Check if device has already been claimed. This may
+	 * happen with driver loading, device discovery/registration,
+	 * and deferred probe processing happens all at once with
+	 * multiple threads.
+	 */
+	if (dev->driver)
+		return -EBUSY;
 
 	if (!driver_match_device(drv, dev))
 		return 0;
 
+	async_allowed = driver_allows_async_probing(drv);
+
+	if (async_allowed)
+		data->have_async = true;
+
+	if (data->check_async && async_allowed != data->want_async)
+		return 0;
+
 	return driver_probe_device(drv, dev);
 }
 
-/**
- * device_attach - try to attach device to a driver.
- * @dev: device.
- *
- * Walk the list of drivers that the bus has and call
- * driver_probe_device() for each pair. If a compatible
- * pair is found, break out and return.
- *
- * Returns 1 if the device was bound to a driver;
- * 0 if no matching driver was found;
- * -ENODEV if the device is not registered.
- *
- * When called for a USB interface, @dev->parent lock must be held.
- */
-int device_attach(struct device *dev)
+static void __device_attach_async_helper(void *_dev, async_cookie_t cookie)
+{
+	struct device *dev = _dev;
+	struct device_attach_data data = {
+		.dev		= dev,
+		.check_async	= true,
+		.want_async	= true,
+	};
+
+	device_lock(dev);
+
+	bus_for_each_drv(dev->bus, NULL, &data, __device_attach_driver);
+	dev_dbg(dev, "async probe completed\n");
+
+	pm_request_idle(dev);
+
+	device_unlock(dev);
+
+	put_device(dev);
+}
+
+int __device_attach(struct device *dev, bool allow_async)
 {
 	int ret = 0;
 
@@ -444,15 +508,59 @@ int device_attach(struct device *dev)
 			ret = 0;
 		}
 	} else {
-		ret = bus_for_each_drv(dev->bus, NULL, dev, __device_attach);
-		pm_request_idle(dev);
+		struct device_attach_data data = {
+			.dev = dev,
+			.check_async = allow_async,
+			.want_async = false,
+		};
+
+		ret = bus_for_each_drv(dev->bus, NULL, &data,
+					__device_attach_driver);
+		if (!ret && allow_async && data.have_async) {
+			/*
+			 * If we could not find appropriate driver
+			 * synchronously and we are allowed to do
+			 * async probes and there are drivers that
+			 * want to probe asynchronously, we'll
+			 * try them.
+			 */
+			dev_dbg(dev, "scheduling asynchronous probe\n");
+			get_device(dev);
+			async_schedule(__device_attach_async_helper, dev);
+		} else {
+			pm_request_idle(dev);
+		}
 	}
 out_unlock:
 	device_unlock(dev);
 	return ret;
 }
+
+/**
+ * device_attach - try to attach device to a driver.
+ * @dev: device.
+ *
+ * Walk the list of drivers that the bus has and call
+ * driver_probe_device() for each pair. If a compatible
+ * pair is found, break out and return.
+ *
+ * Returns 1 if the device was bound to a driver;
+ * 0 if no matching driver was found;
+ * -ENODEV if the device is not registered.
+ *
+ * When called for a USB interface, @dev->parent lock must be held.
+ */
+int device_attach(struct device *dev)
+{
+	return __device_attach(dev, false);
+}
 EXPORT_SYMBOL_GPL(device_attach);
 
+void device_initial_probe(struct device *dev)
+{
+	__device_attach(dev, true);
+}
+
 static int __driver_attach(struct device *dev, void *data)
 {
 	struct device_driver *drv = data;
@@ -507,6 +615,9 @@ static void __device_release_driver(struct device *dev)
 
 	drv = dev->driver;
 	if (drv) {
+		if (driver_allows_async_probing(drv))
+			async_synchronize_full();
+
 		pm_runtime_get_sync(dev);
 
 		driver_sysfs_remove(dev);
diff --git a/include/linux/device.h b/include/linux/device.h
index fb50673..9ace4c0 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -195,6 +195,25 @@ extern struct kset *bus_get_kset(struct bus_type *bus);
 extern struct klist *bus_get_device_klist(struct bus_type *bus);
 
 /**
+  * enum probe_type - device driver probe type to try
+  *	Device drivers may opt in for special handling of their
+  *	respective probe routines. This tells the core what to
+  *	expect and prefer.
+  *
+  * @PROBE_SYNCHRONOUS: Default. Drivers expect their probe routines
+  *	to run synchronously with driver and device registration
+  *	(with the exception of -EPROBE_DEFER handling - re-probing
+  *	always ends up being done asynchronously).
+  * @PROBE_PREFER_ASYNCHRONOUS: Drivers for "slow" devices which
+  *	probing order is not essential for booting the system may
+  *	opt into executing their probes asynchronously.
+  */
+enum probe_type {
+	PROBE_SYNCHRONOUS,
+	PROBE_PREFER_ASYNCHRONOUS,
+};
+
+/**
  * struct device_driver - The basic device driver structure
  * @name:	Name of the device driver.
  * @bus:	The bus which the device of this driver belongs to.
@@ -234,6 +253,7 @@ struct device_driver {
 	const char		*mod_name;	/* used for built-in modules */
 
 	bool suppress_bind_attrs;	/* disables bind/unbind via sysfs */
+	enum probe_type probe_type;
 
 	const struct of_device_id	*of_match_table;
 	const struct acpi_device_id	*acpi_match_table;
@@ -972,6 +992,7 @@ extern int __must_check device_bind_driver(struct device *dev);
 extern void device_release_driver(struct device *dev);
 extern int  __must_check device_attach(struct device *dev);
 extern int __must_check driver_attach(struct device_driver *drv);
+extern void device_initial_probe(struct device *dev);
 extern int __must_check device_reprobe(struct device *dev);
 
 /*
-- 
2.2.0.rc0.207.ga3a616c