[PATCH 1/2] async: async device driver probing

[PATCH 1/2] async: async device driver probing

[falcon]

From: Wu Zhangjin <wuzhangjin@gmail.com>

[*Note*: NOT applicable, only for comments.]

To async the slow driver probing function of some devices, the device probing
support is modified to support async scheduling.

In order to async your driver probing function, please mask the async_probe
flag to 1, and to make sure one asynced probing is executed before an specified
point, please call async_synchronize_full() in that point..

Usage:

	static struct i2c_driver test_driver = {
		.driver = {
			.name   = TEST_DEV_NAME,
			.owner  = THIS_MODULE,
	+               .async_probe = 1,
		},

Signed-off-by: Wu Zhangjin <falcon@meizu.com>
---
 drivers/base/dd.c      |   36 +++++++++++++++++++++++++++++++++---
 include/linux/device.h |    2 ++
 2 files changed, 35 insertions(+), 3 deletions(-)

diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index 0605176..357f36e 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -23,6 +23,7 @@
 #include <linux/kthread.h>
 #include <linux/wait.h>
 #include <linux/async.h>
+#include <linux/slab.h>
 #include <linux/pm_runtime.h>
 #include <linux/pinctrl/devinfo.h>
 
@@ -357,6 +358,11 @@ void wait_for_device_probe(void)
 }
 EXPORT_SYMBOL_GPL(wait_for_device_probe);
 
+struct stupid_thread_structure {
+	struct device_driver *drv;
+	struct device *dev;
+};
+
 /**
  * driver_probe_device - attempt to bind device & driver together
  * @drv: driver to bind a device to
@@ -368,8 +374,23 @@ EXPORT_SYMBOL_GPL(wait_for_device_probe);
  * This function must be called with @dev lock held.  When called for a
  * USB interface, @dev->parent lock must be held as well.
  */
+static void __driver_probe_device(void *void_data, async_cookie_t cookie)
+{
+	struct stupid_thread_structure *data = void_data;
+	struct device_driver *drv = data->drv;
+	struct device *dev = data->dev;
+
+	pm_runtime_barrier(dev);
+	really_probe(dev, drv);
+	pm_request_idle(dev);
+
+	kfree(data);
+}
+
 int driver_probe_device(struct device_driver *drv, struct device *dev)
 {
+	struct stupid_thread_structure *data;
+	async_cookie_t cookie;
 	int ret = 0;
 
 	if (!device_is_registered(dev))
@@ -378,9 +399,18 @@ int driver_probe_device(struct device_driver *drv, struct device *dev)
 	pr_debug("bus: '%s': %s: matched device %s with driver %s\n",
 		 drv->bus->name, __func__, dev_name(dev), drv->name);
 
-	pm_runtime_barrier(dev);
-	ret = really_probe(dev, drv);
-	pm_request_idle(dev);
+	if (drv->async_probe) {
+		data = kmalloc(sizeof(*data), GFP_KERNEL);
+		data->drv = drv;
+		data->dev = dev;
+
+		cookie = async_schedule(__driver_probe_device, data);
+		pr_info("%s: async call %s driver, cookie is %llu\n", __func__, drv->name, cookie);
+	} else {
+		pm_runtime_barrier(dev);
+		ret = really_probe(dev, drv);
+		pm_request_idle(dev);
+	}
 
 	return ret;
 }
diff --git a/include/linux/device.h b/include/linux/device.h
index 952b010..f39ee48 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -247,6 +247,8 @@ struct device_driver {
 	const struct dev_pm_ops *pm;
 
 	struct driver_private *p;
+
+	unsigned int async_probe:1;
 };
 
 
-- 
1.7.10.4

[PATCH 2/2] async: Allow to group the asynced device probings

[falcon]

From: Wu Zhangjin <falcon@meizu.com>

[*Note*: NOT applicable, only for comments.]

This allows to schedule a group of probings in order.

Usage:

If the probing of driver2 depends on the probing of driver1, we can put them
into a group, here put them into a group named domain 1, they will be probed in
the linking order.

...

static struct platform_driver first_driver = {
 	.probe = first_driver_probe,
 	.driver		= {
 		.name	= "first driver",
+		.async_probe = 1,
+		.async_domain = 1,
 	},
 };

...

static struct platform_driver second_driver = {
 	.probe = second_driver_probe,
 	.driver		= {
 		.name	= "second_driver",
+		.async_probe = 1,
+		.async_domain = 1,
 	},
 };

...

With this feature, it is possible to async different class of drivers, for
example, put all sound drivers into domain 2, put all display/video drivers
into domain 3, and sensors domain 4, network drivers domain 5 and so forth.

*TODO*:

 o To share the existing wait_for_device_probe(), register all async domains
 with registered=1. But it is too early than our wait_for_async_probe_domain().

 o It may be possible to async the whole kernel initcalls(except the one
 before scheduler available) with more complicated group features, currently,
 this implementation only allows to group the probings linearly, but the real
 dependencies of the probings are more complicated, to solve this issue, group
 the probings in a *tree* architecture may work.

Signed-off-by: Wu Zhangjin <falcon@meizu.com>
---
 drivers/base/dd.c      |  101 ++++++++++++++++++++++++++++++++++++++++++++++--
 include/linux/device.h |    1 +
 init/main.c            |    4 ++
 3 files changed, 103 insertions(+), 3 deletions(-)

diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index 357f36e..025d8a9 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -358,11 +358,52 @@ void wait_for_device_probe(void)
 }
 EXPORT_SYMBOL_GPL(wait_for_device_probe);
 
+struct async_domain_structure {
+	/* Data Elements */
+	struct async_domain domain;
+	unsigned int domain_id;
+	unsigned int domain_members;
+	struct completion domain_order;
+	char *last_drv;
+	/* List Link */
+	struct list_head list;
+};
+
 struct stupid_thread_structure {
 	struct device_driver *drv;
 	struct device *dev;
+	struct async_domain_structure *domain;
+	unsigned int wait:1;
 };
 
+static LIST_HEAD(async_domain_list);
+
+/* Create the domain list based on the ids, the same ids share the same domain.  */
+struct async_domain_structure *get_async_domain(unsigned int domain_id)
+{
+	struct async_domain_structure *domain;
+
+	/* Check if our list exist, If exist, return it */
+	if (!list_empty(&async_domain_list)) {
+		list_for_each_entry(domain, &async_domain_list, list)
+			if (domain->domain_id == domain_id) {
+				domain->domain_members++;
+				return domain;
+			}
+	}
+
+	/* If not exist, add a new one  */
+	domain = kzalloc(sizeof(struct async_domain_structure), GFP_KERNEL);
+	domain->domain_id = domain_id;
+	domain->domain_members = 1;
+	init_completion(&domain->domain_order);
+	INIT_LIST_HEAD(&domain->domain.pending);
+	domain->domain.registered = 0;
+	list_add(&domain->list, &async_domain_list);
+
+	return domain;
+}
+
 /**
  * driver_probe_device - attempt to bind device & driver together
  * @drv: driver to bind a device to
@@ -379,12 +420,57 @@ static void __driver_probe_device(void *void_data, async_cookie_t cookie)
 	struct stupid_thread_structure *data = void_data;
 	struct device_driver *drv = data->drv;
 	struct device *dev = data->dev;
+	struct async_domain_structure *domain = data->domain;
+	unsigned int wait = data->wait;
+
+	/* Wait for the previous one */
+	if (wait) {
+		pr_info("%s: %s: wait for %s\n", __func__, drv->name, (char *)domain->last_drv);
+		wait_for_completion_interruptible(&domain->domain_order);
+	}
+
+	if (domain)
+		domain->last_drv = (char *)drv->name;
 
 	pm_runtime_barrier(dev);
 	really_probe(dev, drv);
 	pm_request_idle(dev);
 
 	kfree(data);
+
+	/* Finish */
+	if (domain) {
+		pr_info("%s: %s: complete device probing\n", __func__, drv->name);
+		complete(&domain->domain_order);
+	}
+}
+
+void wait_for_async_probe_in_domain(void)
+{
+	struct async_domain_structure *domain;
+
+	pr_info("%s: Wait for all asynced probe devices\n", __func__);
+	list_for_each_entry(domain, &async_domain_list, list) {
+		pr_debug("%s: Wait for domain %d\n", __func__, domain->domain_id);
+		async_synchronize_full_domain(&domain->domain);
+		pr_debug("%s: Release the data struct for domain %d\n", __func__, domain->domain_id);
+		kfree(domain);
+	}
+}
+
+static inline void async_probe_in_domain(void *void_data)
+{
+	struct stupid_thread_structure *data = void_data;
+	struct device_driver *drv = data->drv;
+	async_cookie_t cookie;
+	struct async_domain_structure *domain;
+
+	/* Schedule the device in the specified domain */
+	domain = data->domain;
+	data->wait = (domain->domain_members > 1) ? 1 : 0;
+	pr_debug("%s: %s: members = %d, wait = %d\n", __func__, drv->name, domain->domain_members, data->wait);
+	cookie = async_schedule_domain(__driver_probe_device, data, &domain->domain);
+	pr_info("%s: async call %s driver, cookie is %llu, domain is %d\n", __func__, drv->name, cookie, drv->async_domain);
 }
 
 int driver_probe_device(struct device_driver *drv, struct device *dev)
@@ -400,13 +486,22 @@ int driver_probe_device(struct device_driver *drv, struct device *dev)
 		 drv->bus->name, __func__, dev_name(dev), drv->name);
 
 	if (drv->async_probe) {
-		data = kmalloc(sizeof(*data), GFP_KERNEL);
+		data = kzalloc(sizeof(struct stupid_thread_structure), GFP_KERNEL);
 		data->drv = drv;
 		data->dev = dev;
 
-		cookie = async_schedule(__driver_probe_device, data);
-		pr_info("%s: async call %s driver, cookie is %llu\n", __func__, drv->name, cookie);
+		if (!drv->async_domain) {
+			data->domain = NULL;
+			cookie = async_schedule(__driver_probe_device, data);
+			pr_info("%s: async call %s driver, cookie is %llu\n", __func__, drv->name, cookie);
+		} else {
+			/* Probe the device with domain specified */
+			pr_info("%s: async_probe_in_domain() %s\n", __func__, drv->name);
+			data->domain = get_async_domain(drv->async_domain);
+			async_probe_in_domain(data);
+		}
 	} else {
+		pr_debug("%s: Probe %s\n", __func__, drv->name);
 		pm_runtime_barrier(dev);
 		ret = really_probe(dev, drv);
 		pm_request_idle(dev);
diff --git a/include/linux/device.h b/include/linux/device.h
index f39ee48..3806947 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -248,6 +248,7 @@ struct device_driver {
 
 	struct driver_private *p;
 
+	unsigned int async_domain;
 	unsigned int async_probe:1;
 };
 
diff --git a/init/main.c b/init/main.c
index febc511..4271b7b 100644
--- a/init/main.c
+++ b/init/main.c
@@ -836,6 +836,8 @@ static int try_to_run_init_process(const char *init_filename)
 
 static noinline void __init kernel_init_freeable(void);
 
+extern void wait_for_async_probe_in_domain(void);
+
 static int __ref kernel_init(void *unused)
 {
 	int ret;
@@ -843,6 +845,8 @@ static int __ref kernel_init(void *unused)
 	kernel_init_freeable();
 	/* need to finish all async __init code before freeing the memory */
 	async_synchronize_full();
+	wait_for_async_probe_in_domain();
+
 	free_initmem();
 	mark_rodata_ro();
 	system_state = SYSTEM_RUNNING;
-- 
1.7.10.4

Re: [PATCH 2/2] async: Allow to group the asynced device probings

[Greg KH]

On Sat, Feb 08, 2014 at 07:05:39PM +0800, falcon@meizu.com wrote:
> From: Wu Zhangjin <falcon@meizu.com>
> 
> [*Note*: NOT applicable, only for comments.]
> 
> This allows to schedule a group of probings in order.
> 
> Usage:
> 
> If the probing of driver2 depends on the probing of driver1, we can put them
> into a group, here put them into a group named domain 1, they will be probed in
> the linking order.
> 
> ...
> 
> static struct platform_driver first_driver = {
>  	.probe = first_driver_probe,
>  	.driver		= {
>  		.name	= "first driver",
> +		.async_probe = 1,
> +		.async_domain = 1,
>  	},
>  };
> 
> ...
> 
> static struct platform_driver second_driver = {
>  	.probe = second_driver_probe,
>  	.driver		= {
>  		.name	= "second_driver",
> +		.async_probe = 1,
> +		.async_domain = 1,
>  	},
>  };
> 
> ...
> 
> With this feature, it is possible to async different class of drivers, for
> example, put all sound drivers into domain 2, put all display/video drivers
> into domain 3, and sensors domain 4, network drivers domain 5 and so forth.

That sounds like a recipie for massive confusion.

Again, what problem is this trying to solve?

greg k-h

Re: [PATCH 1/2] async: async device driver probing

[Greg KH]

On Sat, Feb 08, 2014 at 07:05:38PM +0800, falcon@meizu.com wrote:
> From: Wu Zhangjin <wuzhangjin@gmail.com>
> 
> [*Note*: NOT applicable, only for comments.]
> 
> To async the slow driver probing function of some devices, the device probing
> support is modified to support async scheduling.
> 
> In order to async your driver probing function, please mask the async_probe
> flag to 1, and to make sure one asynced probing is executed before an specified
> point, please call async_synchronize_full() in that point..
> 
> Usage:
> 
> 	static struct i2c_driver test_driver = {
> 		.driver = {
> 			.name   = TEST_DEV_NAME,
> 			.owner  = THIS_MODULE,
> 	+               .async_probe = 1,
> 		},

Why is this needed, we have defered probing and the container stuff, so
what problem is this solving?

greg k-h

Re: [PATCH 1/2] async: async device driver probing

[Dmitry Torokhov]

Hi Greg,

On Sat, Feb 08, 2014 at 10:27:29AM -0800, Greg KH wrote:
> On Sat, Feb 08, 2014 at 07:05:38PM +0800, falcon@meizu.com wrote:
> > From: Wu Zhangjin <wuzhangjin@gmail.com>
> > 
> > [*Note*: NOT applicable, only for comments.]
> > 
> > To async the slow driver probing function of some devices, the device probing
> > support is modified to support async scheduling.
> > 
> > In order to async your driver probing function, please mask the async_probe
> > flag to 1, and to make sure one asynced probing is executed before an specified
> > point, please call async_synchronize_full() in that point..
> > 
> > Usage:
> > 
> > 	static struct i2c_driver test_driver = {
> > 		.driver = {
> > 			.name   = TEST_DEV_NAME,
> > 			.owner  = THIS_MODULE,
> > 	+               .async_probe = 1,
> > 		},
> 
> Why is this needed, we have defered probing and the container stuff, so
> what problem is this solving?

Deferred probing only helps if resources are not ready yet, but
sometimes we have a slow(ish) device which initialization stalls probing
the rest of the system. For example a touchpad can take up to a second
to calibrate itself after reset. One could try scheduling reset
asynchronously, or try to offload it to open(), but that is not always
best:

1. Manual async: what to do when reset fails? Ideally we do not want to
leave driver half-way bound with device not operable, but much rather
signal the rest of the system that binding of the device failed and
release all resources.

2. Offload to open: the same issue as with manually doing async reset,
plus sometimes we do not know all parameters that we should create input
device with until after we reset physical device and queried it for
capabilities.

Marking a driver to tell device core to execute probe asynchronously [at
boot time] seems like a very appealing feature from driver author POV.

What is the container stuff you mention?

Thanks.

-- 
Dmitry

Re: [PATCH 1/2] async: async device driver probing

[Greg KH]

On Wed, Aug 13, 2014 at 10:20:33AM -0700, Dmitry Torokhov wrote:
> Hi Greg,
> 
> On Sat, Feb 08, 2014 at 10:27:29AM -0800, Greg KH wrote:

February?  That's an old thread to dig up...


> > On Sat, Feb 08, 2014 at 07:05:38PM +0800, falcon@meizu.com wrote:
> > > From: Wu Zhangjin <wuzhangjin@gmail.com>
> > > 
> > > [*Note*: NOT applicable, only for comments.]
> > > 
> > > To async the slow driver probing function of some devices, the device probing
> > > support is modified to support async scheduling.
> > > 
> > > In order to async your driver probing function, please mask the async_probe
> > > flag to 1, and to make sure one asynced probing is executed before an specified
> > > point, please call async_synchronize_full() in that point..
> > > 
> > > Usage:
> > > 
> > > 	static struct i2c_driver test_driver = {
> > > 		.driver = {
> > > 			.name   = TEST_DEV_NAME,
> > > 			.owner  = THIS_MODULE,
> > > 	+               .async_probe = 1,
> > > 		},
> > 
> > Why is this needed, we have defered probing and the container stuff, so
> > what problem is this solving?
> 
> Deferred probing only helps if resources are not ready yet, but
> sometimes we have a slow(ish) device which initialization stalls probing
> the rest of the system. For example a touchpad can take up to a second
> to calibrate itself after reset. One could try scheduling reset
> asynchronously, or try to offload it to open(), but that is not always
> best:
> 
> 1. Manual async: what to do when reset fails? Ideally we do not want to
> leave driver half-way bound with device not operable, but much rather
> signal the rest of the system that binding of the device failed and
> release all resources.
> 
> 2. Offload to open: the same issue as with manually doing async reset,
> plus sometimes we do not know all parameters that we should create input
> device with until after we reset physical device and queried it for
> capabilities.
> 
> Marking a driver to tell device core to execute probe asynchronously [at
> boot time] seems like a very appealing feature from driver author POV.
> 
> What is the container stuff you mention?

drivers/base/container.c

Re: [PATCH 1/2] async: async device driver probing

[Dmitry Torokhov]

On Thu, Aug 14, 2014 at 06:02:23AM +0800, Greg KH wrote:
> On Wed, Aug 13, 2014 at 10:20:33AM -0700, Dmitry Torokhov wrote:
> > Hi Greg,
> > 
> > On Sat, Feb 08, 2014 at 10:27:29AM -0800, Greg KH wrote:
> 
> February?  That's an old thread to dig up...

Well, yes ;)

> 
> 
> > > On Sat, Feb 08, 2014 at 07:05:38PM +0800, falcon@meizu.com wrote:
> > > > From: Wu Zhangjin <wuzhangjin@gmail.com>
> > > > 
> > > > [*Note*: NOT applicable, only for comments.]
> > > > 
> > > > To async the slow driver probing function of some devices, the device probing
> > > > support is modified to support async scheduling.
> > > > 
> > > > In order to async your driver probing function, please mask the async_probe
> > > > flag to 1, and to make sure one asynced probing is executed before an specified
> > > > point, please call async_synchronize_full() in that point..
> > > > 
> > > > Usage:
> > > > 
> > > > 	static struct i2c_driver test_driver = {
> > > > 		.driver = {
> > > > 			.name   = TEST_DEV_NAME,
> > > > 			.owner  = THIS_MODULE,
> > > > 	+               .async_probe = 1,
> > > > 		},
> > > 
> > > Why is this needed, we have defered probing and the container stuff, so
> > > what problem is this solving?
> > 
> > Deferred probing only helps if resources are not ready yet, but
> > sometimes we have a slow(ish) device which initialization stalls probing
> > the rest of the system. For example a touchpad can take up to a second
> > to calibrate itself after reset. One could try scheduling reset
> > asynchronously, or try to offload it to open(), but that is not always
> > best:
> > 
> > 1. Manual async: what to do when reset fails? Ideally we do not want to
> > leave driver half-way bound with device not operable, but much rather
> > signal the rest of the system that binding of the device failed and
> > release all resources.
> > 
> > 2. Offload to open: the same issue as with manually doing async reset,
> > plus sometimes we do not know all parameters that we should create input
> > device with until after we reset physical device and queried it for
> > capabilities.
> > 
> > Marking a driver to tell device core to execute probe asynchronously [at
> > boot time] seems like a very appealing feature from driver author POV.
> > 
> > What is the container stuff you mention?
> 
> drivers/base/container.c

I am not sure how that would help in the scenario described above, as it
seems ACPI specific...

Thanks.

-- 
Dmitry

Re: [PATCH 1/2] async: async device driver probing

[Greg KH]

On Wed, Aug 13, 2014 at 03:10:44PM -0700, Dmitry Torokhov wrote:
> On Thu, Aug 14, 2014 at 06:02:23AM +0800, Greg KH wrote:
> > On Wed, Aug 13, 2014 at 10:20:33AM -0700, Dmitry Torokhov wrote:
> > > Hi Greg,
> > > 
> > > On Sat, Feb 08, 2014 at 10:27:29AM -0800, Greg KH wrote:
> > 
> > February?  That's an old thread to dig up...
> 
> Well, yes ;)
> 
> > 
> > 
> > > > On Sat, Feb 08, 2014 at 07:05:38PM +0800, falcon@meizu.com wrote:
> > > > > From: Wu Zhangjin <wuzhangjin@gmail.com>
> > > > > 
> > > > > [*Note*: NOT applicable, only for comments.]
> > > > > 
> > > > > To async the slow driver probing function of some devices, the device probing
> > > > > support is modified to support async scheduling.
> > > > > 
> > > > > In order to async your driver probing function, please mask the async_probe
> > > > > flag to 1, and to make sure one asynced probing is executed before an specified
> > > > > point, please call async_synchronize_full() in that point..
> > > > > 
> > > > > Usage:
> > > > > 
> > > > > 	static struct i2c_driver test_driver = {
> > > > > 		.driver = {
> > > > > 			.name   = TEST_DEV_NAME,
> > > > > 			.owner  = THIS_MODULE,
> > > > > 	+               .async_probe = 1,
> > > > > 		},
> > > > 
> > > > Why is this needed, we have defered probing and the container stuff, so
> > > > what problem is this solving?
> > > 
> > > Deferred probing only helps if resources are not ready yet, but
> > > sometimes we have a slow(ish) device which initialization stalls probing
> > > the rest of the system. For example a touchpad can take up to a second
> > > to calibrate itself after reset. One could try scheduling reset
> > > asynchronously, or try to offload it to open(), but that is not always
> > > best:
> > > 
> > > 1. Manual async: what to do when reset fails? Ideally we do not want to
> > > leave driver half-way bound with device not operable, but much rather
> > > signal the rest of the system that binding of the device failed and
> > > release all resources.
> > > 
> > > 2. Offload to open: the same issue as with manually doing async reset,
> > > plus sometimes we do not know all parameters that we should create input
> > > device with until after we reset physical device and queried it for
> > > capabilities.
> > > 
> > > Marking a driver to tell device core to execute probe asynchronously [at
> > > boot time] seems like a very appealing feature from driver author POV.
> > > 
> > > What is the container stuff you mention?
> > 
> > drivers/base/container.c
> 
> I am not sure how that would help in the scenario described above, as it
> seems ACPI specific...

It is?  I thought the drm people were using it on ARM.  I really don't
know, sorry, don't have the chance to check it right now...

greg k-h

Re: [PATCH 1/2] async: async device driver probing

[Greg KH]

On Sun, Feb 09, 2014 at 08:40:43AM +0000, 吴章金 wrote:
> Hi, Greg
> 
> These two patches aim to thread the initcalls(only probes here) for SMP
> systems. to simplify the upgrade of Linux for Android smartphones, the modules
> are built into the Linux kernel image, and hence the initcalls are called in
> serial order, to speedup the booting, benefit from SMP, thread these initcalls
> may parallel the probings.
> 
> and I rechecked the deferred probing, seems it solves the problem of getting
> devices initialized in the right order, the probing will be threaded only if
> the probe fails with the -EPROBE_DEFER. it doesn't provide any mechanism to
> thread a probe explicitly.

It's up to the bus to do threaded probing, but watch out, that is almost
never the bottleneck in your boot process (hint, in Android, booting the
kernel is almost not even in the range that can be measured overall).
There are so many other things that Android does at boot time that could
be worked on instead of worrying about this :)

good luck,

greg k-h