[RFC/RFT][PATCH v2 0/7] Functional dependencies between devices

[RFC/RFT][PATCH v2 0/7] Functional dependencies between devices

[Rafael J. Wysocki]

Hi Everyone,

This is a refresh of the functional dependencies series that I posted last
year and which has picked up by Marek quite recently.  For reference, appended
is my introductory message sent previously (which may be slightly outdated now).

As last time, the first patch rearranges the code around __device_release_driver()
a bit to prepare it for the next one (it actually hasn't changed AFAICS).

The second patch introduces the actual device links mechanics, but without
system suspend/resume and runtime PM support which are added by the subsequent
patches.

Some bugs found by Marek during his work on these patches should be fixed
here.  In particular, the endless recursion in device_reorder_to_tail()
which simply was broken before.

There are two additional patches to address the issue with runtime PM support
that occured when runtime PM was disabled for some suppliers due to a PM
sleep transition in progress.  Those patches simply make runtime PM helpers
return 0 in that case which may be controversial, so please let me know if
there are concerns about those.

The way device_link_add() works is a bit different, as it takes an additional
status argument now.  That makes it possible to create a link in any state,
with extra care of course, and should address the problem pointed to by Lukas
during the previous discussion.

Also some comments from Tomeu have been addressed.

This hasn't been really tested yet and I'm sort of relying on Marek to test
it, because he has a use case ready.  Hence, the RFT tag on the series.

Overall, please let me know what you think.

Thanks,
Rafael


Introduction:

As discussed in the recent "On-demand device probing" thread and in a Kernel
Summit session earlier today, there is a problem with handling cases where
functional dependencies between devices are involved.

What I mean by a "functional dependency" is when the driver of device B needs
both device A and its driver to be present and functional to be able to work.
This implies that the driver of A needs to be working for B to be probed
successfully and it cannot be unbound from the device before the B's driver.
This also has certain consequences for power management of these devices
(suspend/resume and runtime PM ordering).

So I want to be able to represent those functional dependencies between devices
and I'd like the driver core to track them and act on them in certain cases
where they matter.  The argument for doing that in the driver core is that
there are quite a few distinct use cases related to that, they are relatively
hard to get right in a driver (if one wants to address all of them properly)
and it only gets worse if multiplied by the number of drivers potentially
needing to do it.  Morever, at least one case (asynchronous system suspend/resume)
cannot be handled in a single driver at all, because it requires the driver of A
to wait for B to suspend (during system suspend) and the driver of B to wait for
A to resume (during system resume).

My idea is to represent a supplier-consumer dependency between devices (or
more precisely between device+driver combos) as a "link" object containing
pointers to the devices in question, a list node for each of them and some
additional information related to the management of those objects, ie.
something like:

struct device_link {
        struct device *supplier;
        struct list_head supplier_node;
        struct device *consumer;
        struct list_head consumer_node;
        <flags, status etc>
};

In general, there will be two lists of those things per device, one list
of links to consumers and one list of links to suppliers.

In that picture, links will be created by calling, say:

int device_add_link(struct device *me, struct device *my_supplier, unsigned int flags);

and they will be deleted by the driver core when not needed any more.  The
creation of a link should also cause dpm_list and the list used during shutdown
to be reordered if needed.

In principle, it seems usefult to consider two types of links, one created
at device registration time (when registering the second device from the linked
pair, whichever it is) and one created at probe time (of the consumer device).
I'll refer to them as "permanent" and "probe-time" links, respectively.

The permanent links (created at device registration time) will stay around
until one of the linked devices is unregistered (at which time the driver
core will drop the link along with the device going away).  The probe-time
ones will be dropped (automatically) at the consumer device driver unbind time.

There's a question about what if the supplier device is being unbound before
the consumer one (for example, as a result of a hotplug event).  My current
view on that is that the consumer needs to be force-unbound in that case too,
but I guess I may be persuaded otherwise given sufficiently convincing
arguments.  Anyway, there are reasons to do that, like for example it may
help with the synchronization.  Namely, if there's a rule that suppliers
cannot be unbound before any consumers linked to them, than the list of links
to suppliers for a consumer can only change at its registration/probe or
unbind/remove times (which simplifies things quite a bit).

With that, the permanent links existing at the probe time for a consumer
device can be used to check whether or not to defer the probing of it
even before executing its probe callback.  In turn, system suspend
synchronization should be a matter of calling device_pm_wait_for_dev()
for all consumers of a supplier device, in analogy with dpm_wait_for_children(),
and so on.

Of course, the new lists have to be stable during those operations and ensuring
that is going to be somewhat tricky (AFAICS right now at least), but apart from
that the whole concept looks reasonably straightforward to me.

[RFC/RFT][PATCH v2 1/7] driver core: Add a wrapper around __device_release_driver()

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Add an internal wrapper around __device_release_driver() that will
acquire device locks and do the necessary checks before calling it.

A subsequent change will make use of it.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/base/dd.c | 30 ++++++++++++++++++------------
 1 file changed, 18 insertions(+), 12 deletions(-)

diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index 16688f50729c..d9e76e9205c7 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -796,6 +796,22 @@ static void __device_release_driver(struct device *dev)
 	}
 }
 
+static void device_release_driver_internal(struct device *dev,
+					   struct device_driver *drv,
+					   struct device *parent)
+{
+	if (parent)
+		device_lock(parent);
+
+	device_lock(dev);
+	if (!drv || drv == dev->driver)
+		__device_release_driver(dev);
+
+	device_unlock(dev);
+	if (parent)
+		device_unlock(parent);
+}
+
 /**
  * device_release_driver - manually detach device from driver.
  * @dev: device.
@@ -810,9 +826,7 @@ void device_release_driver(struct device *dev)
 	 * within their ->remove callback for the same device, they
 	 * will deadlock right here.
 	 */
-	device_lock(dev);
-	__device_release_driver(dev);
-	device_unlock(dev);
+	device_release_driver_internal(dev, NULL, NULL);
 }
 EXPORT_SYMBOL_GPL(device_release_driver);
 
@@ -837,15 +851,7 @@ void driver_detach(struct device_driver *drv)
 		dev = dev_prv->device;
 		get_device(dev);
 		spin_unlock(&drv->p->klist_devices.k_lock);
-
-		if (dev->parent)	/* Needed for USB */
-			device_lock(dev->parent);
-		device_lock(dev);
-		if (dev->driver == drv)
-			__device_release_driver(dev);
-		device_unlock(dev);
-		if (dev->parent)
-			device_unlock(dev->parent);
+		device_release_driver_internal(dev, drv, dev->parent);
 		put_device(dev);
 	}
 }

[RFC/RFT][PATCH v2 2/7] driver core: Functional dependencies tracking support

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Currently, there is a problem with handling cases where functional
dependencies between devices are involved.

What I mean by a "functional dependency" is when the driver of device
B needs both device A and its driver to be present and functional to
be able to work.  This implies that the driver of A needs to be
working for B to be probed successfully and it cannot be unbound from
the device before the B's driver.  This also has certain consequences
for power management of these devices (suspend/resume and runtime PM
ordering).

Add support for representing those functional dependencies between
devices to allow the driver core to track them and act on them in
certain cases where they matter.

The argument for doing that in the driver core is that there are
quite a few distinct use cases related to that, they are relatively
hard to get right in a driver (if one wants to address all of them
properly) and it only gets worse if multiplied by the number of
drivers potentially needing to do it.  Morever, at least one case
(asynchronous system suspend/resume) cannot be handled in a single
driver at all, because it requires the driver of A to wait for B to
suspend (during system suspend) and the driver of B to wait for
A to resume (during system resume).

To that end, represent links between devices (or more precisely
between device+driver combos) as a struct device_link object
containing pointers to the devices in question, a list node for
each of them, status information, flags, a lock and an RCU head
for synchronization.

Also add two new list heads, links_to_consumers and links_to_suppliers,
to struct device to represent the lists of links to the devices that
depend on the given one (consumers) and to the devices depended on
by it (suppliers), respectively.

The entire data structure consisting of all of the lists of link
objects for all devices is protected by SRCU (for list walking)
and a by mutex (for link object addition/removal).  In addition
to that, each link object has an internal status field whose
value reflects what's happening to the devices pointed to by
the link.  That status field is protected by an internal spinlock.

New links are added by calling device_link_add() which may happen
either before the consumer device is probed or when probing it, in
which case the caller must ensure that the driver of the supplier
device is present and functional and the DEVICE_LINK_SUPPLIER_READY
flag must be passed to device_link_add() to reflect that.

Link objects are deleted either explicitly, by calling
device_link_del() on the link object in question, or automatically,
when the consumer device is unbound from its driver or when one
of the target devices is deleted, depending on the link type.

There are two types of link objects, persistent and non-persistent.
The persistent ones stay around until one of the target devices is
deleted, while the non-persistent ones are deleted when the consumer
driver is unbound from its device (ie. they are assumed to be valid
only as long as the consumer device has a driver bound to it).  The
DEVICE_LINK_PERSISTENT flag is passed to device_link_add() to create
a persistent link and it cannot be used for links created at the
consumer probe time (that is, persistent links must be created before
probing the consumer devices).

One of the actions carried out by device_link_add() is to reorder
the lists used for device shutdown and system suspend/resume to
put the consumer device along with all of its children and all of
its consumers (and so on, recursively) to the ends of those list
in order to ensure the right ordering between the all of the supplier
and consumer devices.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/base/base.h    |   11 +
 drivers/base/core.c    |  421 +++++++++++++++++++++++++++++++++++++++++++++++++
 drivers/base/dd.c      |   42 ++++
 include/linux/device.h |   35 ++++
 4 files changed, 504 insertions(+), 5 deletions(-)

Index: linux-pm/drivers/base/base.h
===================================================================
--- linux-pm.orig/drivers/base/base.h
+++ linux-pm/drivers/base/base.h
@@ -107,6 +107,9 @@ extern void bus_remove_device(struct dev
 
 extern int bus_add_driver(struct device_driver *drv);
 extern void bus_remove_driver(struct device_driver *drv);
+extern void device_release_driver_internal(struct device *dev,
+					   struct device_driver *drv,
+					   struct device *parent);
 
 extern void driver_detach(struct device_driver *drv);
 extern int driver_probe_device(struct device_driver *drv, struct device *dev);
@@ -152,3 +155,11 @@ extern int devtmpfs_init(void);
 #else
 static inline int devtmpfs_init(void) { return 0; }
 #endif
+
+/* Device links */
+extern int device_links_check_suppliers(struct device *dev);
+extern void device_links_driver_bound(struct device *dev);
+extern void device_links_driver_gone(struct device *dev);
+extern void device_links_no_driver(struct device *dev);
+extern bool device_links_busy(struct device *dev);
+extern void device_links_unbind_consumers(struct device *dev);
Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -44,6 +44,402 @@ static int __init sysfs_deprecated_setup
 early_param("sysfs.deprecated", sysfs_deprecated_setup);
 #endif
 
+/* Device links support. */
+
+DEFINE_STATIC_SRCU(device_links_srcu);
+static DEFINE_MUTEX(device_links_lock);
+
+/**
+ * device_is_dependent - Check if one device depends on another one
+ * @dev: Device to check dependencies for.
+ * @target: Device to check against.
+ *
+ * Check if @dev or any device dependent on it (its child or its consumer etc)
+ * depends on @target.  Return 1 if that is the case or 0 otherwise.
+ */
+static int device_is_dependent(struct device *dev, void *target)
+{
+	struct device_link *link;
+	int ret;
+
+	ret = device_for_each_child(dev, target, device_is_dependent);
+	list_for_each_entry(link, &dev->links_to_consumers, s_node) {
+		if (WARN_ON(link->consumer == target))
+			return 1;
+
+		ret = ret || device_is_dependent(link->consumer, target);
+	}
+	return ret;
+}
+
+static int device_reorder_to_tail(struct device *dev, void *not_used)
+{
+	struct device_link *link;
+
+	devices_kset_move_last(dev);
+	device_pm_move_last(dev);
+	device_for_each_child(dev, NULL, device_reorder_to_tail);
+	list_for_each_entry(link, &dev->links_to_consumers, s_node)
+		device_reorder_to_tail(link->consumer, NULL);
+
+	return 0;
+}
+
+/**
+ * device_link_add - Create a link between two devices.
+ * @consumer: Consumer end of the link.
+ * @supplier: Supplier end of the link.
+ * @status: Initial status of the link.
+ * @flags: Link flags.
+ *
+ * The caller is responsible for ensuring that (a) @status reflects the current
+ * status of both @consumer and @supplier and (b) the creation of the new link
+ * is properly synchronized with runtime PM (especially if @status is
+ * DEVICE_LINK_ACTIVE).
+ *
+ * A side effect of the link creation is re-ordering of dpm_list and the
+ * devices_kset list by moving the consumer device and all devices depending
+ * on it to the ends of these lists.
+ */
+struct device_link *device_link_add(struct device *consumer,
+				    struct device *supplier,
+				    enum device_link_status status, u32 flags)
+{
+	struct device_link *link;
+
+	if (!consumer || !supplier || status == DEVICE_LINK_SUPPLIER_UNBIND)
+		return NULL;
+
+	mutex_lock(&device_links_lock);
+
+	/*
+	 * If there is a reverse dependency between the consumer and the
+	 * supplier already in the graph, return NULL.
+	 */
+	if (device_is_dependent(consumer, supplier)) {
+		link = NULL;
+		goto out;
+	}
+
+	list_for_each_entry(link, &supplier->links_to_consumers, s_node)
+		if (link->consumer == consumer)
+			goto out;
+
+	link = kmalloc(sizeof(*link), GFP_KERNEL);
+	if (!link)
+		goto out;
+
+	get_device(supplier);
+	link->supplier = supplier;
+	INIT_LIST_HEAD(&link->s_node);
+	get_device(consumer);
+	link->consumer = consumer;
+	INIT_LIST_HEAD(&link->c_node);
+	spin_lock_init(&link->lock);
+	link->flags = flags;
+	link->status = status;
+
+	/*
+	 * Move the consumer and all of the devices depending on it to the end
+	 * of dpm_list and the devices_kset list.
+	 *
+	 * It is necessary to hold dpm_list locked throughout all that or else
+	 * we may end up suspending with a wrong ordering of it.
+	 */
+	device_pm_lock();
+	device_reorder_to_tail(consumer, NULL);
+	device_pm_unlock();
+
+	list_add_tail_rcu(&link->s_node, &supplier->links_to_consumers);
+	list_add_tail_rcu(&link->c_node, &consumer->links_to_suppliers);
+
+	dev_info(consumer, "Linked as a consumer to %s\n", dev_name(supplier));
+
+ out:
+	mutex_unlock(&device_links_lock);
+	return link;
+}
+EXPORT_SYMBOL_GPL(device_link_add);
+
+static void __device_link_free_srcu(struct rcu_head *rhead)
+{
+	struct device_link *link;
+
+	link = container_of(rhead, struct device_link, rcu_head);
+	put_device(link->consumer);
+	put_device(link->supplier);
+	kfree(link);
+}
+
+static void __device_link_del(struct device_link *link)
+{
+	dev_info(link->consumer, "Dropping the link to %s\n",
+		 dev_name(link->supplier));
+
+	list_del_rcu(&link->s_node);
+	list_del_rcu(&link->c_node);
+	call_srcu(&device_links_srcu, &link->rcu_head, __device_link_free_srcu);
+}
+
+/**
+ * device_link_del - Delete a link between two devices.
+ * @link: Device link to delete.
+ *
+ * The caller must ensure proper synchronization of this function with runtime
+ * PM.
+ */
+void device_link_del(struct device_link *link)
+{
+	mutex_lock(&device_links_lock);
+	device_pm_lock();
+	__device_link_del(link);
+	device_pm_unlock();
+	mutex_unlock(&device_links_lock);
+}
+EXPORT_SYMBOL_GPL(device_link_del);
+
+static int device_links_read_lock(void)
+{
+	return srcu_read_lock(&device_links_srcu);
+}
+
+static void device_links_read_unlock(int idx)
+{
+	return srcu_read_unlock(&device_links_srcu, idx);
+}
+
+static void device_links_missing_supplier(struct device *dev)
+{
+	struct device_link *link;
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
+		spin_lock(&link->lock);
+
+		if (link->status == DEVICE_LINK_CONSUMER_PROBE)
+			link->status = DEVICE_LINK_AVAILABLE;
+
+		spin_unlock(&link->lock);
+	}
+}
+
+/**
+ * device_links_check_suppliers - Check supplier devices for this one.
+ * @dev: Consumer device.
+ *
+ * Check links from this device to any suppliers.  Walk the list of the device's
+ * consumer links and see if all of the suppliers are available.  If not, simply
+ * return -EPROBE_DEFER.
+ *
+ * Walk the list under SRCU and check each link's status field under its lock.
+ *
+ * We need to guarantee that the supplier will not go away after the check has
+ * been positive here.  It only can go away in __device_release_driver() and
+ * that function  checks the device's links to consumers.  This means we need to
+ * mark the link as "consumer probe in progress" to make the supplier removal
+ * wait for us to complete (or bad things may happen).
+ */
+int device_links_check_suppliers(struct device *dev)
+{
+	struct device_link *link;
+	int idx, ret = 0;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
+		spin_lock(&link->lock);
+		if (link->status != DEVICE_LINK_AVAILABLE) {
+			spin_unlock(&link->lock);
+			device_links_missing_supplier(dev);
+			ret = -EPROBE_DEFER;
+			break;
+		}
+		link->status = DEVICE_LINK_CONSUMER_PROBE;
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+	return ret;
+}
+
+/**
+ * device_links_driver_bound - Update device links after probing its driver.
+ * @dev: Device to update the links for.
+ *
+ * The probe has been successful, so update links from this device to any
+ * consumers by changing their status to "available".
+ *
+ * Also change the status of @dev's links to suppliers to "active".
+ */
+void device_links_driver_bound(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
+		spin_lock(&link->lock);
+		WARN_ON(link->status != DEVICE_LINK_DORMANT);
+		link->status = DEVICE_LINK_AVAILABLE;
+		spin_unlock(&link->lock);
+	}
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
+		spin_lock(&link->lock);
+		WARN_ON(link->status != DEVICE_LINK_CONSUMER_PROBE);
+		link->status = DEVICE_LINK_ACTIVE;
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+}
+
+/**
+ * device_links_driver_gone - Update links after driver removal.
+ * @dev: Device whose driver has gone away.
+ *
+ * Update links to consumers for @dev by changing their status to "dormant".
+ */
+void device_links_driver_gone(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
+		WARN_ON(!(link->flags & DEVICE_LINK_PERSISTENT));
+		spin_lock(&link->lock);
+		WARN_ON(link->status != DEVICE_LINK_SUPPLIER_UNBIND);
+		link->status = DEVICE_LINK_DORMANT;
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+}
+
+/**
+ * device_links_no_driver - Update links of a device without a driver.
+ * @dev: Device without a drvier.
+ *
+ * Delete all non-persistent links from this device to any suppliers.
+ * Persistent links stay around, but their status is changed to "available",
+ * unless they already are in the "supplier unbind in progress" state in which
+ * case they need not be updated.
+ */
+void device_links_no_driver(struct device *dev)
+{
+	struct device_link *link, *ln;
+
+	mutex_lock(&device_links_lock);
+
+	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_suppliers, c_node)
+		if (link->flags & DEVICE_LINK_PERSISTENT) {
+			spin_lock(&link->lock);
+
+			if (link->status != DEVICE_LINK_SUPPLIER_UNBIND)
+				link->status = DEVICE_LINK_AVAILABLE;
+
+			spin_unlock(&link->lock);
+		} else {
+			__device_link_del(link);
+		}
+
+	mutex_unlock(&device_links_lock);
+}
+
+/**
+ * device_links_busy - Check if there are any busy links to consumers.
+ * @dev: Device to check.
+ *
+ * Check each consumer of the device and return 'true' if its link's status
+ * is one of "consumer probe" or "active" (meaning that the given consumer is
+ * probing right now or its driver is present).  Otherwise, change the link
+ * state to "supplier unbind" to prevent the consumer from being probed
+ * successfully going forward.
+ *
+ * Return 'false' if there are no probing or active consumers.
+ */
+bool device_links_busy(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+	bool ret = false;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
+		spin_lock(&link->lock);
+		if (link->status == DEVICE_LINK_CONSUMER_PROBE
+		    || link->status == DEVICE_LINK_ACTIVE) {
+			spin_unlock(&link->lock);
+			ret = true;
+			break;
+		}
+		link->status = DEVICE_LINK_SUPPLIER_UNBIND;
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+	return ret;
+}
+
+/**
+ * device_links_unbind_consumers - Force unbind consumers of the given device.
+ * @dev: Device to unbind the consumers of.
+ *
+ * Walk the list of links to consumers for @dev and if any of them is in the
+ * "consumer probe" state, wait for all device probes in progress to complete
+ * and start over.
+ *
+ * If that's not the case, change the status of the link to "supplier unbind"
+ * and check if the link was in the "active" state.  If so, force the consumer
+ * driver to unbind and start over (the consumer will not re-probe as we have
+ * changed the state of the link already).
+ */
+void device_links_unbind_consumers(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+ start:
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
+		enum device_link_status status;
+
+		spin_lock(&link->lock);
+		status = link->status;
+		if (status == DEVICE_LINK_CONSUMER_PROBE) {
+			spin_unlock(&link->lock);
+
+			device_links_read_unlock(idx);
+
+			wait_for_device_probe();
+			goto start;
+		}
+		link->status = DEVICE_LINK_SUPPLIER_UNBIND;
+		if (status == DEVICE_LINK_ACTIVE) {
+			struct device *consumer = link->consumer;
+
+			get_device(consumer);
+			spin_unlock(&link->lock);
+
+			device_links_read_unlock(idx);
+
+			device_release_driver_internal(consumer, NULL,
+						       consumer->parent);
+			put_device(consumer);
+			goto start;
+		}
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+}
+
+/* Device links support end. */
+
 int (*platform_notify)(struct device *dev) = NULL;
 int (*platform_notify_remove)(struct device *dev) = NULL;
 static struct kobject *dev_kobj;
@@ -711,6 +1107,8 @@ void device_initialize(struct device *de
 #ifdef CONFIG_GENERIC_MSI_IRQ
 	INIT_LIST_HEAD(&dev->msi_list);
 #endif
+	INIT_LIST_HEAD(&dev->links_to_consumers);
+	INIT_LIST_HEAD(&dev->links_to_suppliers);
 }
 EXPORT_SYMBOL_GPL(device_initialize);
 
@@ -1233,6 +1631,7 @@ void device_del(struct device *dev)
 {
 	struct device *parent = dev->parent;
 	struct class_interface *class_intf;
+	struct device_link *link, *ln;
 
 	/* Notify clients of device removal.  This call must come
 	 * before dpm_sysfs_remove().
@@ -1240,6 +1639,28 @@ void device_del(struct device *dev)
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DEL_DEVICE, dev);
+
+	/*
+	 * Delete all of the remaining links from this device to any other
+	 * devices (either consumers or suppliers).
+	 *
+	 * This requires that all links be dormant, so warn if that's no the
+	 * case.
+	 */
+	mutex_lock(&device_links_lock);
+
+	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_suppliers, c_node) {
+		WARN_ON(link->status != DEVICE_LINK_DORMANT);
+		__device_link_del(link);
+	}
+
+	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_consumers, s_node) {
+		WARN_ON(link->status != DEVICE_LINK_DORMANT);
+		__device_link_del(link);
+	}
+
+	mutex_unlock(&device_links_lock);
+
 	dpm_sysfs_remove(dev);
 	if (parent)
 		klist_del(&dev->p->knode_parent);
Index: linux-pm/drivers/base/dd.c
===================================================================
--- linux-pm.orig/drivers/base/dd.c
+++ linux-pm/drivers/base/dd.c
@@ -249,6 +249,7 @@ static void driver_bound(struct device *
 		 __func__, dev_name(dev));
 
 	klist_add_tail(&dev->p->knode_driver, &dev->driver->p->klist_devices);
+	device_links_driver_bound(dev);
 
 	device_pm_check_callbacks(dev);
 
@@ -399,6 +400,7 @@ probe_failed:
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
 pinctrl_bind_failed:
+	device_links_no_driver(dev);
 	devres_release_all(dev);
 	driver_sysfs_remove(dev);
 	dev->driver = NULL;
@@ -489,6 +491,10 @@ int driver_probe_device(struct device_dr
 	if (!device_is_registered(dev))
 		return -ENODEV;
 
+	ret = device_links_check_suppliers(dev);
+	if (ret)
+		return ret;
+
 	pr_debug("bus: '%s': %s: matched device %s with driver %s\n",
 		 drv->bus->name, __func__, dev_name(dev), drv->name);
 
@@ -756,7 +762,7 @@ EXPORT_SYMBOL_GPL(driver_attach);
  * __device_release_driver() must be called with @dev lock held.
  * When called for a USB interface, @dev->parent lock must be held as well.
  */
-static void __device_release_driver(struct device *dev)
+static void __device_release_driver(struct device *dev, struct device *parent)
 {
 	struct device_driver *drv;
 
@@ -765,6 +771,25 @@ static void __device_release_driver(stru
 		if (driver_allows_async_probing(drv))
 			async_synchronize_full();
 
+		while (device_links_busy(dev)) {
+			device_unlock(dev);
+			if (parent)
+				device_unlock(parent);
+
+			device_links_unbind_consumers(dev);
+			if (parent)
+				device_lock(parent);
+
+			device_lock(dev);
+			/*
+			 * A concurrent invocation of the same function might
+			 * have released the driver successfully while this one
+			 * was waiting, so check for that.
+			 */
+			if (dev->driver != drv)
+				return;
+		}
+
 		pm_runtime_get_sync(dev);
 
 		driver_sysfs_remove(dev);
@@ -780,6 +805,9 @@ static void __device_release_driver(stru
 			dev->bus->remove(dev);
 		else if (drv->remove)
 			drv->remove(dev);
+
+		device_links_driver_gone(dev);
+		device_links_no_driver(dev);
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);
@@ -796,16 +824,16 @@ static void __device_release_driver(stru
 	}
 }
 
-static void device_release_driver_internal(struct device *dev,
-					   struct device_driver *drv,
-					   struct device *parent)
+void device_release_driver_internal(struct device *dev,
+				    struct device_driver *drv,
+				    struct device *parent)
 {
 	if (parent)
 		device_lock(parent);
 
 	device_lock(dev);
 	if (!drv || drv == dev->driver)
-		__device_release_driver(dev);
+		__device_release_driver(dev, parent);
 
 	device_unlock(dev);
 	if (parent)
@@ -818,6 +846,10 @@ static void device_release_driver_intern
  *
  * Manually detach device from driver.
  * When called for a USB interface, @dev->parent lock must be held.
+ *
+ * If this function is to be called with @dev->parent lock held, ensure that
+ * the device's consumers are unbound in advance or that their locks can be
+ * acquired under the @dev->parent lock.
  */
 void device_release_driver(struct device *dev)
 {
Index: linux-pm/include/linux/device.h
===================================================================
--- linux-pm.orig/include/linux/device.h
+++ linux-pm/include/linux/device.h
@@ -706,6 +706,32 @@ struct device_dma_parameters {
 	unsigned long segment_boundary_mask;
 };
 
+enum device_link_status {
+	DEVICE_LINK_DORMANT = 0,	/* Link not in use. */
+	DEVICE_LINK_AVAILABLE,		/* Supplier driver is present. */
+	DEVICE_LINK_ACTIVE,		/* Consumer driver is present too. */
+	DEVICE_LINK_CONSUMER_PROBE,	/* Consumer is probing. */
+	DEVICE_LINK_SUPPLIER_UNBIND,	/* Supplier is unbinding. */
+};
+
+/*
+ * Device link flags.
+ *
+ * PERSISTENT: Do not delete the link on consumer device driver unbind.
+ */
+#define DEVICE_LINK_PERSISTENT	(1 << 0)
+
+struct device_link {
+	struct device *supplier;
+	struct list_head s_node;
+	struct device *consumer;
+	struct list_head c_node;
+	enum device_link_status status;
+	u32 flags;
+	spinlock_t lock;
+	struct rcu_head rcu_head;
+};
+
 /**
  * struct device - The basic device structure
  * @parent:	The device's "parent" device, the device to which it is attached.
@@ -731,6 +757,8 @@ struct device_dma_parameters {
  * 		on.  This shrinks the "Board Support Packages" (BSPs) and
  * 		minimizes board-specific #ifdefs in drivers.
  * @driver_data: Private pointer for driver specific info.
+ * @links_to_consumers: Links to consumer devices.
+ * @links_to_suppliers: Links to supplier devices.
  * @power:	For device power management.
  * 		See Documentation/power/devices.txt for details.
  * @pm_domain:	Provide callbacks that are executed during system suspend,
@@ -797,6 +825,8 @@ struct device {
 					   core doesn't touch it */
 	void		*driver_data;	/* Driver data, set and get with
 					   dev_set/get_drvdata */
+	struct list_head	links_to_consumers;
+	struct list_head	links_to_suppliers;
 	struct dev_pm_info	power;
 	struct dev_pm_domain	*pm_domain;
 
@@ -1113,6 +1143,11 @@ extern void device_shutdown(void);
 /* debugging and troubleshooting/diagnostic helpers. */
 extern const char *dev_driver_string(const struct device *dev);
 
+/* Device links interface. */
+struct device_link *device_link_add(struct device *consumer,
+				    struct device *supplier,
+				    enum device_link_status status, u32 flags);
+void device_link_del(struct device_link *link);
 
 #ifdef CONFIG_PRINTK
 

Re: [RFC/RFT][PATCH v2 2/7] driver core: Functional dependencies tracking support

[Ulf Hansson]

+ Mark

On 8 September 2016 at 23:27, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
>
> Currently, there is a problem with handling cases where functional
> dependencies between devices are involved.
>
> What I mean by a "functional dependency" is when the driver of device
> B needs both device A and its driver to be present and functional to
> be able to work.  This implies that the driver of A needs to be
> working for B to be probed successfully and it cannot be unbound from
> the device before the B's driver.  This also has certain consequences
> for power management of these devices (suspend/resume and runtime PM
> ordering).
>
> Add support for representing those functional dependencies between
> devices to allow the driver core to track them and act on them in
> certain cases where they matter.
>
> The argument for doing that in the driver core is that there are
> quite a few distinct use cases related to that, they are relatively
> hard to get right in a driver (if one wants to address all of them
> properly) and it only gets worse if multiplied by the number of
> drivers potentially needing to do it.  Morever, at least one case
> (asynchronous system suspend/resume) cannot be handled in a single
> driver at all, because it requires the driver of A to wait for B to
> suspend (during system suspend) and the driver of B to wait for
> A to resume (during system resume).
>
> To that end, represent links between devices (or more precisely
> between device+driver combos) as a struct device_link object
> containing pointers to the devices in question, a list node for
> each of them, status information, flags, a lock and an RCU head
> for synchronization.
>
> Also add two new list heads, links_to_consumers and links_to_suppliers,
> to struct device to represent the lists of links to the devices that
> depend on the given one (consumers) and to the devices depended on
> by it (suppliers), respectively.
>
> The entire data structure consisting of all of the lists of link
> objects for all devices is protected by SRCU (for list walking)
> and a by mutex (for link object addition/removal).  In addition
> to that, each link object has an internal status field whose
> value reflects what's happening to the devices pointed to by
> the link.  That status field is protected by an internal spinlock.
>
> New links are added by calling device_link_add() which may happen
> either before the consumer device is probed or when probing it, in
> which case the caller must ensure that the driver of the supplier
> device is present and functional and the DEVICE_LINK_SUPPLIER_READY
> flag must be passed to device_link_add() to reflect that.
>
> Link objects are deleted either explicitly, by calling
> device_link_del() on the link object in question, or automatically,
> when the consumer device is unbound from its driver or when one
> of the target devices is deleted, depending on the link type.
>
> There are two types of link objects, persistent and non-persistent.
> The persistent ones stay around until one of the target devices is
> deleted, while the non-persistent ones are deleted when the consumer
> driver is unbound from its device (ie. they are assumed to be valid
> only as long as the consumer device has a driver bound to it).  The
> DEVICE_LINK_PERSISTENT flag is passed to device_link_add() to create
> a persistent link and it cannot be used for links created at the
> consumer probe time (that is, persistent links must be created before
> probing the consumer devices).
>
> One of the actions carried out by device_link_add() is to reorder
> the lists used for device shutdown and system suspend/resume to
> put the consumer device along with all of its children and all of
> its consumers (and so on, recursively) to the ends of those list
> in order to ensure the right ordering between the all of the supplier
> and consumer devices.

Rafael, thanks for working on this and re-spinning this series. It's
indeed very interesting!

I am hoping "device links" should be able to solve some of those
device ordering issues I have observed for several SoCs, particularly
during system PM and in combination with runtime PM.

I intend to test the series as soon as I can and try to deploy it to
see if it solves some of the issues I have seen. I will also try to
review in more detail. No promises short term though. :-)

BTW, as I am mostly working on DT based platforms, I guess we would
later on need to discuss with the DT maintainers how to describe
device links.

A minor comment to the change-log. I would appreciate some information
about "error" handling. Especially, what happens in the driver core
when it's about to probe a device with a configured device link, but
the link hasn’t been established yet (the other device isn't
successfully probed). In the ideal scenario this shouldn't happen, but
of course it will. So I assume the driver core relies on the existing
deferred probe mechanism for this.

[...]

Kind regards
Uffe

Re: [RFC/RFT][PATCH v2 2/7] driver core: Functional dependencies tracking support

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 488 bytes --]

On Fri, Sep 09, 2016 at 10:25:30AM +0200, Ulf Hansson wrote:

> BTW, as I am mostly working on DT based platforms, I guess we would
> later on need to discuss with the DT maintainers how to describe
> device links.

I think the expectation had been that since DT tends to get right down
to the hardware details we already have the information in there in
order to use things so the problem becomes figuring out how/when we
parse the existing bindings to tell the driver core about links.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [RFC/RFT][PATCH v2 2/7] driver core: Functional dependencies tracking support

[Ulf Hansson]

On 9 September 2016 at 14:06, Mark Brown <broonie@kernel.org> wrote:
> On Fri, Sep 09, 2016 at 10:25:30AM +0200, Ulf Hansson wrote:
>
>> BTW, as I am mostly working on DT based platforms, I guess we would
>> later on need to discuss with the DT maintainers how to describe
>> device links.
>
> I think the expectation had been that since DT tends to get right down
> to the hardware details we already have the information in there in
> order to use things so the problem becomes figuring out how/when we
> parse the existing bindings to tell the driver core about links.

I guess what we need from a functional point of view, is to be able to
first initialize all devices (device_initialize()) then add the
relevant device links before we add the devices (device_add()). That
seems like a piece of cake to fix. :-)

Kind regards
Uffe

Re: [RFC/RFT][PATCH v2 2/7] driver core: Functional dependencies tracking support

[Rafael J. Wysocki]

On Friday, September 09, 2016 10:25:30 AM Ulf Hansson wrote:
> + Mark
> 
> On 8 September 2016 at 23:27, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> >
> > Currently, there is a problem with handling cases where functional
> > dependencies between devices are involved.
> >
> > What I mean by a "functional dependency" is when the driver of device
> > B needs both device A and its driver to be present and functional to
> > be able to work.  This implies that the driver of A needs to be
> > working for B to be probed successfully and it cannot be unbound from
> > the device before the B's driver.  This also has certain consequences
> > for power management of these devices (suspend/resume and runtime PM
> > ordering).
> >
> > Add support for representing those functional dependencies between
> > devices to allow the driver core to track them and act on them in
> > certain cases where they matter.
> >
> > The argument for doing that in the driver core is that there are
> > quite a few distinct use cases related to that, they are relatively
> > hard to get right in a driver (if one wants to address all of them
> > properly) and it only gets worse if multiplied by the number of
> > drivers potentially needing to do it.  Morever, at least one case
> > (asynchronous system suspend/resume) cannot be handled in a single
> > driver at all, because it requires the driver of A to wait for B to
> > suspend (during system suspend) and the driver of B to wait for
> > A to resume (during system resume).
> >
> > To that end, represent links between devices (or more precisely
> > between device+driver combos) as a struct device_link object
> > containing pointers to the devices in question, a list node for
> > each of them, status information, flags, a lock and an RCU head
> > for synchronization.
> >
> > Also add two new list heads, links_to_consumers and links_to_suppliers,
> > to struct device to represent the lists of links to the devices that
> > depend on the given one (consumers) and to the devices depended on
> > by it (suppliers), respectively.
> >
> > The entire data structure consisting of all of the lists of link
> > objects for all devices is protected by SRCU (for list walking)
> > and a by mutex (for link object addition/removal).  In addition
> > to that, each link object has an internal status field whose
> > value reflects what's happening to the devices pointed to by
> > the link.  That status field is protected by an internal spinlock.
> >
> > New links are added by calling device_link_add() which may happen
> > either before the consumer device is probed or when probing it, in
> > which case the caller must ensure that the driver of the supplier
> > device is present and functional and the DEVICE_LINK_SUPPLIER_READY
> > flag must be passed to device_link_add() to reflect that.
> >
> > Link objects are deleted either explicitly, by calling
> > device_link_del() on the link object in question, or automatically,
> > when the consumer device is unbound from its driver or when one
> > of the target devices is deleted, depending on the link type.
> >
> > There are two types of link objects, persistent and non-persistent.
> > The persistent ones stay around until one of the target devices is
> > deleted, while the non-persistent ones are deleted when the consumer
> > driver is unbound from its device (ie. they are assumed to be valid
> > only as long as the consumer device has a driver bound to it).  The
> > DEVICE_LINK_PERSISTENT flag is passed to device_link_add() to create
> > a persistent link and it cannot be used for links created at the
> > consumer probe time (that is, persistent links must be created before
> > probing the consumer devices).
> >
> > One of the actions carried out by device_link_add() is to reorder
> > the lists used for device shutdown and system suspend/resume to
> > put the consumer device along with all of its children and all of
> > its consumers (and so on, recursively) to the ends of those list
> > in order to ensure the right ordering between the all of the supplier
> > and consumer devices.
> 
> Rafael, thanks for working on this and re-spinning this series. It's
> indeed very interesting!

Well, you're welcome. :-)

> I am hoping "device links" should be able to solve some of those
> device ordering issues I have observed for several SoCs, particularly
> during system PM and in combination with runtime PM.
> 
> I intend to test the series as soon as I can and try to deploy it to
> see if it solves some of the issues I have seen. I will also try to
> review in more detail. No promises short term though. :-)

All feedback will be appreciated.

I think I'll send an update of it tomorrow, though.

> BTW, as I am mostly working on DT based platforms, I guess we would
> later on need to discuss with the DT maintainers how to describe
> device links.
> 
> A minor comment to the change-log. I would appreciate some information
> about "error" handling. Especially, what happens in the driver core
> when it's about to probe a device with a configured device link, but
> the link hasn’t been established yet (the other device isn't
> successfully probed). In the ideal scenario this shouldn't happen, but
> of course it will. So I assume the driver core relies on the existing
> deferred probe mechanism for this.

Yes, it does.  I'll update the changelog with this information.

Thanks,
Rafael

Re: [RFC/RFT][PATCH v2 2/7] driver core: Functional dependencies tracking support

[Lukas Wunner]

On Thu, Sep 08, 2016 at 11:27:45PM +0200, Rafael J. Wysocki wrote:
> +/**
> + * device_is_dependent - Check if one device depends on another one
> + * @dev: Device to check dependencies for.
> + * @target: Device to check against.
> + *
> + * Check if @dev or any device dependent on it (its child or its consumer etc)
> + * depends on @target.  Return 1 if that is the case or 0 otherwise.
> + */
> +static int device_is_dependent(struct device *dev, void *target)
> +{
> +	struct device_link *link;
> +	int ret;
> +
> +	ret = device_for_each_child(dev, target, device_is_dependent);
> +	list_for_each_entry(link, &dev->links_to_consumers, s_node) {
> +		if (WARN_ON(link->consumer == target))
> +			return 1;
> +
> +		ret = ret || device_is_dependent(link->consumer, target);
> +	}
> +	return ret;
> +}

What happens if someone tries to add a device link from a parent
(as the consumer) to a child (as a supplier)?  You're only checking
if target is a consumer of dev, for full correctness you'd also have
to check if target is a parent of dev.  (Or grandparent, or great-
grandparent, ... you need to walk the tree up to the root.)


The function can be sped up by returning immediately if a match
is found instead of continuing searching and accumulating the
result in ret, i.e.:

	if (device_for_each_child(dev, target, device_is_dependent))
		return 1;

and in the list_for_each_entry block:

	if (device_is_dependent(link->consumer, target))
		return 1;

Then at the end of the function "return 0".


I'd move the WARN_ON() to the single invocation of this function in
device_link_add(), that way it's possible to use the function as a
helper elsewhere should the need arise.

Thanks,

Lukas

Re: [RFC/RFT][PATCH v2 2/7] driver core: Functional dependencies tracking support

[Lukas Wunner]

On Sun, Sep 11, 2016 at 03:40:58PM +0200, Lukas Wunner wrote:
> On Thu, Sep 08, 2016 at 11:27:45PM +0200, Rafael J. Wysocki wrote:
> > +/**
> > + * device_is_dependent - Check if one device depends on another one
> > + * @dev: Device to check dependencies for.
> > + * @target: Device to check against.
> > + *
> > + * Check if @dev or any device dependent on it (its child or its consumer etc)
> > + * depends on @target.  Return 1 if that is the case or 0 otherwise.
> > + */
> > +static int device_is_dependent(struct device *dev, void *target)
> > +{
> > +	struct device_link *link;
> > +	int ret;
> > +
> > +	ret = device_for_each_child(dev, target, device_is_dependent);
> > +	list_for_each_entry(link, &dev->links_to_consumers, s_node) {
> > +		if (WARN_ON(link->consumer == target))
> > +			return 1;
> > +
> > +		ret = ret || device_is_dependent(link->consumer, target);
> > +	}
> > +	return ret;
> > +}
> 
> What happens if someone tries to add a device link from a parent
> (as the consumer) to a child (as a supplier)?  You're only checking
> if target is a consumer of dev, for full correctness you'd also have
> to check if target is a parent of dev.  (Or grandparent, or great-
> grandparent, ... you need to walk the tree up to the root.)
> 
> 
> The function can be sped up by returning immediately if a match
> is found instead of continuing searching and accumulating the
> result in ret, i.e.:
> 
> 	if (device_for_each_child(dev, target, device_is_dependent))
> 		return 1;
> 
> and in the list_for_each_entry block:
> 
> 	if (device_is_dependent(link->consumer, target))
> 		return 1;
> 
> Then at the end of the function "return 0".
> 
> 
> I'd move the WARN_ON() to the single invocation of this function in
> device_link_add(), that way it's possible to use the function as a
> helper elsewhere should the need arise.

Oh I'm grasping only now, you want to emit a WARN for *every*
infringing child/consumer. That could lead to a WARN flood if
a developer accidentally does something really dumb, like linking
the PCI root to some PCI endpoint device, but fair enough.

The point about linking a parent to a child still stands however.
I think a simple way to check this is to just add

	if (WARN_ON(dev == target))
		return 1;

at the top of the function, because when someone tries to link
a parent to a child, when recursing from the parent downward
one will eventually hit that child. This will also prevent
someone from linking a device to itself.

Best regards,

Lukas

Re: [RFC/RFT][PATCH v2 2/7] driver core: Functional dependencies tracking support

[Rafael J. Wysocki]

On Sunday, September 11, 2016 10:43:36 PM Lukas Wunner wrote:
> On Sun, Sep 11, 2016 at 03:40:58PM +0200, Lukas Wunner wrote:
> > On Thu, Sep 08, 2016 at 11:27:45PM +0200, Rafael J. Wysocki wrote:
> > > +/**
> > > + * device_is_dependent - Check if one device depends on another one
> > > + * @dev: Device to check dependencies for.
> > > + * @target: Device to check against.
> > > + *
> > > + * Check if @dev or any device dependent on it (its child or its consumer etc)
> > > + * depends on @target.  Return 1 if that is the case or 0 otherwise.
> > > + */
> > > +static int device_is_dependent(struct device *dev, void *target)
> > > +{
> > > +	struct device_link *link;
> > > +	int ret;
> > > +
> > > +	ret = device_for_each_child(dev, target, device_is_dependent);
> > > +	list_for_each_entry(link, &dev->links_to_consumers, s_node) {
> > > +		if (WARN_ON(link->consumer == target))
> > > +			return 1;
> > > +
> > > +		ret = ret || device_is_dependent(link->consumer, target);
> > > +	}
> > > +	return ret;
> > > +}
> > 
> > What happens if someone tries to add a device link from a parent
> > (as the consumer) to a child (as a supplier)?  You're only checking
> > if target is a consumer of dev, for full correctness you'd also have
> > to check if target is a parent of dev.  (Or grandparent, or great-
> > grandparent, ... you need to walk the tree up to the root.)
> > 
> > 
> > The function can be sped up by returning immediately if a match
> > is found instead of continuing searching and accumulating the
> > result in ret, i.e.:
> > 
> > 	if (device_for_each_child(dev, target, device_is_dependent))
> > 		return 1;
> > 
> > and in the list_for_each_entry block:
> > 
> > 	if (device_is_dependent(link->consumer, target))
> > 		return 1;
> > 
> > Then at the end of the function "return 0".
> > 
> > 
> > I'd move the WARN_ON() to the single invocation of this function in
> > device_link_add(), that way it's possible to use the function as a
> > helper elsewhere should the need arise.
> 
> Oh I'm grasping only now, you want to emit a WARN for *every*
> infringing child/consumer. That could lead to a WARN flood if
> a developer accidentally does something really dumb, like linking
> the PCI root to some PCI endpoint device, but fair enough.
> 
> The point about linking a parent to a child still stands however.
> I think a simple way to check this is to just add
> 
> 	if (WARN_ON(dev == target))
> 		return 1;
> 
> at the top of the function, because when someone tries to link
> a parent to a child, when recursing from the parent downward
> one will eventually hit that child. This will also prevent
> someone from linking a device to itself.

I actually would prefer to make it impossible to link a parent to
a child at all.

Thanks,
Rafael

Re: [RFC/RFT][PATCH v2 2/7] driver core: Functional dependencies tracking support

[Lukas Wunner]

On Wed, Sep 14, 2016 at 03:21:27AM +0200, Rafael J. Wysocki wrote:
> On Sunday, September 11, 2016 10:43:36 PM Lukas Wunner wrote:
> > On Sun, Sep 11, 2016 at 03:40:58PM +0200, Lukas Wunner wrote:
> > > On Thu, Sep 08, 2016 at 11:27:45PM +0200, Rafael J. Wysocki wrote:
> > > > +/**
> > > > + * device_is_dependent - Check if one device depends on another one
> > > > + * @dev: Device to check dependencies for.
> > > > + * @target: Device to check against.
> > > > + *
> > > > + * Check if @dev or any device dependent on it (its child or its consumer etc)
> > > > + * depends on @target.  Return 1 if that is the case or 0 otherwise.
> > > > + */
> > > > +static int device_is_dependent(struct device *dev, void *target)
> > > > +{
> > > > +	struct device_link *link;
> > > > +	int ret;
> > > > +
> > > > +	ret = device_for_each_child(dev, target, device_is_dependent);
> > > > +	list_for_each_entry(link, &dev->links_to_consumers, s_node) {
> > > > +		if (WARN_ON(link->consumer == target))
> > > > +			return 1;
> > > > +
> > > > +		ret = ret || device_is_dependent(link->consumer, target);
> > > > +	}
> > > > +	return ret;
> > > > +}
> > > 
> > > What happens if someone tries to add a device link from a parent
> > > (as the consumer) to a child (as a supplier)?  You're only checking
> > > if target is a consumer of dev, for full correctness you'd also have
> > > to check if target is a parent of dev.  (Or grandparent, or great-
> > > grandparent, ... you need to walk the tree up to the root.)
> > > 
> > > 
> > > The function can be sped up by returning immediately if a match
> > > is found instead of continuing searching and accumulating the
> > > result in ret, i.e.:
> > > 
> > > 	if (device_for_each_child(dev, target, device_is_dependent))
> > > 		return 1;
> > > 
> > > and in the list_for_each_entry block:
> > > 
> > > 	if (device_is_dependent(link->consumer, target))
> > > 		return 1;
> > > 
> > > Then at the end of the function "return 0".
> > > 
> > > 
> > > I'd move the WARN_ON() to the single invocation of this function in
> > > device_link_add(), that way it's possible to use the function as a
> > > helper elsewhere should the need arise.
> > 
> > Oh I'm grasping only now, you want to emit a WARN for *every*
> > infringing child/consumer. That could lead to a WARN flood if
> > a developer accidentally does something really dumb, like linking
> > the PCI root to some PCI endpoint device, but fair enough.
> > 
> > The point about linking a parent to a child still stands however.
> > I think a simple way to check this is to just add
> > 
> > 	if (WARN_ON(dev == target))
> > 		return 1;
> > 
> > at the top of the function, because when someone tries to link
> > a parent to a child, when recursing from the parent downward
> > one will eventually hit that child. This will also prevent
> > someone from linking a device to itself.
> 
> I actually would prefer to make it impossible to link a parent to
> a child at all.

Which is precisely what the code snippet above does.

Thanks,

Lukas

Re: [RFC/RFT][PATCH v2 2/7] driver core: Functional dependencies tracking support

[Rafael J. Wysocki]

On Wednesday, September 14, 2016 10:28:24 AM Lukas Wunner wrote:
> On Wed, Sep 14, 2016 at 03:21:27AM +0200, Rafael J. Wysocki wrote:
> > On Sunday, September 11, 2016 10:43:36 PM Lukas Wunner wrote:
> > > On Sun, Sep 11, 2016 at 03:40:58PM +0200, Lukas Wunner wrote:
> > > > On Thu, Sep 08, 2016 at 11:27:45PM +0200, Rafael J. Wysocki wrote:
> > > > > +/**
> > > > > + * device_is_dependent - Check if one device depends on another one
> > > > > + * @dev: Device to check dependencies for.
> > > > > + * @target: Device to check against.
> > > > > + *
> > > > > + * Check if @dev or any device dependent on it (its child or its consumer etc)
> > > > > + * depends on @target.  Return 1 if that is the case or 0 otherwise.
> > > > > + */
> > > > > +static int device_is_dependent(struct device *dev, void *target)
> > > > > +{
> > > > > +	struct device_link *link;
> > > > > +	int ret;
> > > > > +
> > > > > +	ret = device_for_each_child(dev, target, device_is_dependent);
> > > > > +	list_for_each_entry(link, &dev->links_to_consumers, s_node) {
> > > > > +		if (WARN_ON(link->consumer == target))
> > > > > +			return 1;
> > > > > +
> > > > > +		ret = ret || device_is_dependent(link->consumer, target);
> > > > > +	}
> > > > > +	return ret;
> > > > > +}
> > > > 
> > > > What happens if someone tries to add a device link from a parent
> > > > (as the consumer) to a child (as a supplier)?  You're only checking
> > > > if target is a consumer of dev, for full correctness you'd also have
> > > > to check if target is a parent of dev.  (Or grandparent, or great-
> > > > grandparent, ... you need to walk the tree up to the root.)
> > > > 
> > > > 
> > > > The function can be sped up by returning immediately if a match
> > > > is found instead of continuing searching and accumulating the
> > > > result in ret, i.e.:
> > > > 
> > > > 	if (device_for_each_child(dev, target, device_is_dependent))
> > > > 		return 1;
> > > > 
> > > > and in the list_for_each_entry block:
> > > > 
> > > > 	if (device_is_dependent(link->consumer, target))
> > > > 		return 1;
> > > > 
> > > > Then at the end of the function "return 0".
> > > > 
> > > > 
> > > > I'd move the WARN_ON() to the single invocation of this function in
> > > > device_link_add(), that way it's possible to use the function as a
> > > > helper elsewhere should the need arise.
> > > 
> > > Oh I'm grasping only now, you want to emit a WARN for *every*
> > > infringing child/consumer. That could lead to a WARN flood if
> > > a developer accidentally does something really dumb, like linking
> > > the PCI root to some PCI endpoint device, but fair enough.
> > > 
> > > The point about linking a parent to a child still stands however.
> > > I think a simple way to check this is to just add
> > > 
> > > 	if (WARN_ON(dev == target))
> > > 		return 1;
> > > 
> > > at the top of the function, because when someone tries to link
> > > a parent to a child, when recursing from the parent downward
> > > one will eventually hit that child. This will also prevent
> > > someone from linking a device to itself.
> > 
> > I actually would prefer to make it impossible to link a parent to
> > a child at all.
> 
> Which is precisely what the code snippet above does.

All right, this means I shouldn't reply to email late in the night.  But
at least we seem to be in agreement here.

Thanks,
Rafael

[RFC/RFT][PATCH v2 3/7] PM / sleep: Make async suspend/resume of devices use device links

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Make the device suspend/resume part of the core system
suspend/resume code use device links to ensure that supplier
and consumer devices will be suspended and resumed in the right
order in case of async suspend/resume.

The idea, roughly, is to use dpm_wait() to wait for all consumers
before a supplier device suspend and to wait for all suppliers
before a consumer device resume.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/base/base.h       |    2 +
 drivers/base/core.c       |    4 +-
 drivers/base/power/main.c |   68 +++++++++++++++++++++++++++++++++++++++++-----
 3 files changed, 66 insertions(+), 8 deletions(-)

Index: linux-pm/drivers/base/base.h
===================================================================
--- linux-pm.orig/drivers/base/base.h
+++ linux-pm/drivers/base/base.h
@@ -163,3 +163,5 @@ extern void device_links_driver_gone(str
 extern void device_links_no_driver(struct device *dev);
 extern bool device_links_busy(struct device *dev);
 extern void device_links_unbind_consumers(struct device *dev);
+extern int device_links_read_lock(void);
+extern void device_links_read_unlock(int idx);
Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -198,12 +198,12 @@ void device_link_del(struct device_link
 }
 EXPORT_SYMBOL_GPL(device_link_del);
 
-static int device_links_read_lock(void)
+int device_links_read_lock(void)
 {
 	return srcu_read_lock(&device_links_srcu);
 }
 
-static void device_links_read_unlock(int idx)
+void device_links_read_unlock(int idx)
 {
 	return srcu_read_unlock(&device_links_srcu, idx);
 }
Index: linux-pm/drivers/base/power/main.c
===================================================================
--- linux-pm.orig/drivers/base/power/main.c
+++ linux-pm/drivers/base/power/main.c
@@ -244,6 +244,62 @@ static void dpm_wait_for_children(struct
        device_for_each_child(dev, &async, dpm_wait_fn);
 }
 
+static void dpm_wait_for_suppliers(struct device *dev, bool async)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	/*
+	 * If the supplier goes away right after we've checked the link to it,
+	 * we'll wait for its completion to change the state, but that's fine,
+	 * because the only things that will block as a result are the SRCU
+	 * callbacks freeing the link objects for the links in the list we're
+	 * walking.
+	 */
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node)
+		if (link->status != DEVICE_LINK_DORMANT)
+			dpm_wait(link->supplier, async);
+
+	device_links_read_unlock(idx);
+}
+
+static void dpm_wait_for_superior(struct device *dev, bool async)
+{
+	dpm_wait(dev->parent, async);
+	dpm_wait_for_suppliers(dev, async);
+}
+
+static void dpm_wait_for_consumers(struct device *dev, bool async)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	/*
+	 * The status of a device link can only be changed from "dormant" by a
+	 * probe, but that cannot happen during system suspend/resume.  In
+	 * theory it can change to "dormant" at that time, but then it is
+	 * reasonable to wait for the target device anyway (eg. if it goes
+	 * away, it's better to wait for it to go away completely and then
+	 * continue instead of trying to continue in parallel with its
+	 * unregistration).
+	 */
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node)
+		if (link->status != DEVICE_LINK_DORMANT)
+			dpm_wait(link->consumer, async);
+
+	device_links_read_unlock(idx);
+}
+
+static void dpm_wait_for_subordinate(struct device *dev, bool async)
+{
+	dpm_wait_for_children(dev, async);
+	dpm_wait_for_consumers(dev, async);
+}
+
 /**
  * pm_op - Return the PM operation appropriate for given PM event.
  * @ops: PM operations to choose from.
@@ -488,7 +544,7 @@ static int device_resume_noirq(struct de
 	if (!dev->power.is_noirq_suspended)
 		goto Out;
 
-	dpm_wait(dev->parent, async);
+	dpm_wait_for_superior(dev, async);
 
 	if (dev->pm_domain) {
 		info = "noirq power domain ";
@@ -618,7 +674,7 @@ static int device_resume_early(struct de
 	if (!dev->power.is_late_suspended)
 		goto Out;
 
-	dpm_wait(dev->parent, async);
+	dpm_wait_for_superior(dev, async);
 
 	if (dev->pm_domain) {
 		info = "early power domain ";
@@ -750,7 +806,7 @@ static int device_resume(struct device *
 		goto Complete;
 	}
 
-	dpm_wait(dev->parent, async);
+	dpm_wait_for_superior(dev, async);
 	dpm_watchdog_set(&wd, dev);
 	device_lock(dev);
 
@@ -1038,7 +1094,7 @@ static int __device_suspend_noirq(struct
 	if (dev->power.syscore || dev->power.direct_complete)
 		goto Complete;
 
-	dpm_wait_for_children(dev, async);
+	dpm_wait_for_subordinate(dev, async);
 
 	if (dev->pm_domain) {
 		info = "noirq power domain ";
@@ -1185,7 +1241,7 @@ static int __device_suspend_late(struct
 	if (dev->power.syscore || dev->power.direct_complete)
 		goto Complete;
 
-	dpm_wait_for_children(dev, async);
+	dpm_wait_for_subordinate(dev, async);
 
 	if (dev->pm_domain) {
 		info = "late power domain ";
@@ -1358,7 +1414,7 @@ static int __device_suspend(struct devic
 	TRACE_DEVICE(dev);
 	TRACE_SUSPEND(0);
 
-	dpm_wait_for_children(dev, async);
+	dpm_wait_for_subordinate(dev, async);
 
 	if (async_error)
 		goto Complete;

Re: [RFC/RFT][PATCH v2 3/7] PM / sleep: Make async suspend/resume of devices use device links

[Lukas Wunner]

On Thu, Sep 08, 2016 at 11:28:33PM +0200, Rafael J. Wysocki wrote:
> Make the device suspend/resume part of the core system
> suspend/resume code use device links to ensure that supplier
> and consumer devices will be suspended and resumed in the right
> order in case of async suspend/resume.
> 
> The idea, roughly, is to use dpm_wait() to wait for all consumers
> before a supplier device suspend and to wait for all suppliers
> before a consumer device resume.

For devices with a parent/child relationship, if the child does not
utilize direct_complete, the parent is not allowed to utilize it
either and is runtime resumed upon system sleep.

Don't we need the same for supplier/consumer relationships?

The code enforcing this is in __device_suspend() and looks like this:

	if (parent) {
		spin_lock_irq(&parent->power.lock);

		dev->parent->power.direct_complete = false;
		if (dev->power.wakeup_path
		    && !dev->parent->power.ignore_children)
			dev->parent->power.wakeup_path = true;

		spin_unlock_irq(&parent->power.lock);
	}

I guess we need to iterate over the suppliers here and execute
the block for each of them.

Thanks,

Lukas

Re: [RFC/RFT][PATCH v2 3/7] PM / sleep: Make async suspend/resume of devices use device links

[Rafael J. Wysocki]

On Sat, Sep 10, 2016 at 3:31 PM, Lukas Wunner <lukas@wunner.de> wrote:
> On Thu, Sep 08, 2016 at 11:28:33PM +0200, Rafael J. Wysocki wrote:
>> Make the device suspend/resume part of the core system
>> suspend/resume code use device links to ensure that supplier
>> and consumer devices will be suspended and resumed in the right
>> order in case of async suspend/resume.
>>
>> The idea, roughly, is to use dpm_wait() to wait for all consumers
>> before a supplier device suspend and to wait for all suppliers
>> before a consumer device resume.
>
> For devices with a parent/child relationship, if the child does not
> utilize direct_complete, the parent is not allowed to utilize it
> either and is runtime resumed upon system sleep.
>
> Don't we need the same for supplier/consumer relationships?
>
> The code enforcing this is in __device_suspend() and looks like this:
>
>         if (parent) {
>                 spin_lock_irq(&parent->power.lock);
>
>                 dev->parent->power.direct_complete = false;
>                 if (dev->power.wakeup_path
>                     && !dev->parent->power.ignore_children)
>                         dev->parent->power.wakeup_path = true;
>
>                 spin_unlock_irq(&parent->power.lock);
>         }
>
> I guess we need to iterate over the suppliers here and execute
> the block for each of them.

You are right about the direct_complete thing, but the wakeup_path
thing is another matter.  It is about forwarding wakeup signals up the
hierarchy and I'd confine it to parents at least for the time being.

Thanks,
Rafael

[RFC/RFT][PATCH v2 4/7] PM / runtime: Pass flags argument to __pm_runtime_disable()

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Modify __pm_runtime_disable() to take a flags argument instead of
the bool one it takes currently which will allow its behavior to
be modified in more than one way.  Introduce a flag
RPM_DISABLE_CHECK_RESUME to address the case currenty addressed by
passing 'true' to __pm_runtime_disable() as the second argument.

A subsequent change will add one more flag to use with
__pm_runtime_disable().

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 Documentation/power/runtime_pm.txt |    2 +-
 drivers/base/power/main.c          |    2 +-
 drivers/base/power/runtime.c       |   10 +++++-----
 include/linux/pm_runtime.h         |    9 ++++++---
 4 files changed, 13 insertions(+), 10 deletions(-)

Index: linux-pm/drivers/base/power/runtime.c
===================================================================
--- linux-pm.orig/drivers/base/power/runtime.c
+++ linux-pm/drivers/base/power/runtime.c
@@ -1158,18 +1158,18 @@ EXPORT_SYMBOL_GPL(pm_runtime_barrier);
 /**
  * __pm_runtime_disable - Disable runtime PM of a device.
  * @dev: Device to handle.
- * @check_resume: If set, check if there's a resume request for the device.
+ * @flags: Behavior modifiers.
  *
  * Increment power.disable_depth for the device and if it was zero previously,
  * cancel all pending runtime PM requests for the device and wait for all
  * operations in progress to complete.  The device can be either active or
  * suspended after its runtime PM has been disabled.
  *
- * If @check_resume is set and there's a resume request pending when
+ * If the CHECK_RESUME flag is set and there's a resume request pending when
  * __pm_runtime_disable() is called and power.disable_depth is zero, the
  * function will wake up the device before disabling its runtime PM.
  */
-void __pm_runtime_disable(struct device *dev, bool check_resume)
+void __pm_runtime_disable(struct device *dev, unsigned int flags)
 {
 	spin_lock_irq(&dev->power.lock);
 
@@ -1183,7 +1183,7 @@ void __pm_runtime_disable(struct device
 	 * means there probably is some I/O to process and disabling runtime PM
 	 * shouldn't prevent the device from processing the I/O.
 	 */
-	if (check_resume && dev->power.request_pending
+	if ((flags & RPM_DISABLE_CHECK_RESUME) && dev->power.request_pending
 	    && dev->power.request == RPM_REQ_RESUME) {
 		/*
 		 * Prevent suspends and idle notifications from being carried
@@ -1442,7 +1442,7 @@ void pm_runtime_reinit(struct device *de
  */
 void pm_runtime_remove(struct device *dev)
 {
-	__pm_runtime_disable(dev, false);
+	__pm_runtime_disable(dev, 0);
 	pm_runtime_reinit(dev);
 }
 
Index: linux-pm/include/linux/pm_runtime.h
===================================================================
--- linux-pm.orig/include/linux/pm_runtime.h
+++ linux-pm/include/linux/pm_runtime.h
@@ -23,6 +23,9 @@
 					    usage_count */
 #define RPM_AUTO		0x08	/* Use autosuspend_delay */
 
+/* Runtime PM disable/enable flags */
+#define RPM_DISABLE_CHECK_RESUME	(1 << 0)
+
 #ifdef CONFIG_PM
 extern struct workqueue_struct *pm_wq;
 
@@ -44,7 +47,7 @@ extern int pm_schedule_suspend(struct de
 extern int __pm_runtime_set_status(struct device *dev, unsigned int status);
 extern int pm_runtime_barrier(struct device *dev);
 extern void pm_runtime_enable(struct device *dev);
-extern void __pm_runtime_disable(struct device *dev, bool check_resume);
+extern void __pm_runtime_disable(struct device *dev, unsigned int flags);
 extern void pm_runtime_allow(struct device *dev);
 extern void pm_runtime_forbid(struct device *dev);
 extern void pm_runtime_no_callbacks(struct device *dev);
@@ -157,7 +160,7 @@ static inline int __pm_runtime_set_statu
 					    unsigned int status) { return 0; }
 static inline int pm_runtime_barrier(struct device *dev) { return 0; }
 static inline void pm_runtime_enable(struct device *dev) {}
-static inline void __pm_runtime_disable(struct device *dev, bool c) {}
+static inline void __pm_runtime_disable(struct device *dev, unsigned int flags) {}
 static inline void pm_runtime_allow(struct device *dev) {}
 static inline void pm_runtime_forbid(struct device *dev) {}
 
@@ -272,7 +275,7 @@ static inline void pm_runtime_set_suspen
 
 static inline void pm_runtime_disable(struct device *dev)
 {
-	__pm_runtime_disable(dev, true);
+	__pm_runtime_disable(dev, RPM_DISABLE_CHECK_RESUME);
 }
 
 static inline void pm_runtime_use_autosuspend(struct device *dev)
Index: linux-pm/drivers/base/power/main.c
===================================================================
--- linux-pm.orig/drivers/base/power/main.c
+++ linux-pm/drivers/base/power/main.c
@@ -1228,7 +1228,7 @@ static int __device_suspend_late(struct
 	TRACE_DEVICE(dev);
 	TRACE_SUSPEND(0);
 
-	__pm_runtime_disable(dev, false);
+	__pm_runtime_disable(dev, 0);
 
 	if (async_error)
 		goto Complete;
Index: linux-pm/Documentation/power/runtime_pm.txt
===================================================================
--- linux-pm.orig/Documentation/power/runtime_pm.txt
+++ linux-pm/Documentation/power/runtime_pm.txt
@@ -685,7 +685,7 @@ out the following operations:
     right before executing the subsystem-level .prepare() callback for it and
     pm_runtime_barrier() is called for every device right before executing the
     subsystem-level .suspend() callback for it.  In addition to that the PM core
-    calls  __pm_runtime_disable() with 'false' as the second argument for every
+    calls  __pm_runtime_disable() with 0 as the second argument for every
     device right before executing the subsystem-level .suspend_late() callback
     for it.
 

[RFC/RFT][PATCH v2 5/7] PM / runtime: Flag to indicate PM sleep transitions in progress

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Introduce a new flag in struct dev_pm_info, pm_sleep_in_progress, to
indicate that runtime PM has been disabled because of a PM sleep
transition in progress.

Make __pm_runtime_disable() set that flag when invoked with
RPM_DISABLE_ENABLE_PM_SLEEP set in its second argument and make
the PM core call it that way during system sleep transitions.

Introduce __pm_runtime_enable() so that it can take a second flags
argument and make it clear power.pm_sleep_in_progress for the device
if invoked with RPM_DISABLE_ENABLE_PM_SLEEP set.  Also make the PM
core pass RPM_DISABLE_ENABLE_PM_SLEEP to it during transitions from
system sleep to the working state.

Modify rpm_idle(), rpm_resume, rpm_suspend() and pm_schedule_suspend()
to neglect error codes and return 0 (without doing anything else)
when power.pm_sleep_in_progress is set for the device.

That will allow helpers like pm_runtime_get_sync() to be called
during system sleep transitions without worrying about possible
error codes they may return because runtime PM is disabled at
that point.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 Documentation/power/runtime_pm.txt |   15 ++++++++-------
 drivers/base/power/main.c          |   12 ++++++------
 drivers/base/power/runtime.c       |   29 +++++++++++++++++++++++------
 include/linux/pm.h                 |    1 +
 include/linux/pm_runtime.h         |   12 ++++++++++--
 5 files changed, 48 insertions(+), 21 deletions(-)

Index: linux-pm/Documentation/power/runtime_pm.txt
===================================================================
--- linux-pm.orig/Documentation/power/runtime_pm.txt
+++ linux-pm/Documentation/power/runtime_pm.txt
@@ -685,14 +685,15 @@ out the following operations:
     right before executing the subsystem-level .prepare() callback for it and
     pm_runtime_barrier() is called for every device right before executing the
     subsystem-level .suspend() callback for it.  In addition to that the PM core
-    calls  __pm_runtime_disable() with 0 as the second argument for every
-    device right before executing the subsystem-level .suspend_late() callback
-    for it.
+    calls  __pm_runtime_disable() with RPM_DISABLE_ENABLE_PM_SLEEP as the second
+    argument for every device right before executing the subsystem-level
+    .suspend_late() callback for it.
 
-  * During system resume pm_runtime_enable() and pm_runtime_put() are called for
-    every device right after executing the subsystem-level .resume_early()
-    callback and right after executing the subsystem-level .complete() callback
-    for it, respectively.
+  * During system resume __pm_runtime_enable() (with RPM_DISABLE_ENABLE_PM_SLEEP
+    as the second argument) and pm_runtime_put() are called for every device
+    right after executing the subsystem-level .resume_early() callback and right
+    after executing the subsystem-level .complete() callback for it,
+    respectively.
 
 7. Generic subsystem callbacks
 
Index: linux-pm/drivers/base/power/main.c
===================================================================
--- linux-pm.orig/drivers/base/power/main.c
+++ linux-pm/drivers/base/power/main.c
@@ -701,7 +701,7 @@ static int device_resume_early(struct de
  Out:
 	TRACE_RESUME(error);
 
-	pm_runtime_enable(dev);
+	__pm_runtime_enable(dev, RPM_DISABLE_ENABLE_PM_SLEEP);
 	complete_all(&dev->power.completion);
 	return error;
 }
@@ -801,8 +801,8 @@ static int device_resume(struct device *
 		goto Complete;
 
 	if (dev->power.direct_complete) {
-		/* Match the pm_runtime_disable() in __device_suspend(). */
-		pm_runtime_enable(dev);
+		/* Match the __pm_runtime_disable() in __device_suspend(). */
+		__pm_runtime_enable(dev, RPM_DISABLE_ENABLE_PM_SLEEP);
 		goto Complete;
 	}
 
@@ -1228,7 +1228,7 @@ static int __device_suspend_late(struct
 	TRACE_DEVICE(dev);
 	TRACE_SUSPEND(0);
 
-	__pm_runtime_disable(dev, 0);
+	__pm_runtime_disable(dev, RPM_DISABLE_ENABLE_PM_SLEEP);
 
 	if (async_error)
 		goto Complete;
@@ -1438,11 +1438,11 @@ static int __device_suspend(struct devic
 
 	if (dev->power.direct_complete) {
 		if (pm_runtime_status_suspended(dev)) {
-			pm_runtime_disable(dev);
+			__pm_runtime_disable(dev, RPM_DISABLE_ALL);
 			if (pm_runtime_status_suspended(dev))
 				goto Complete;
 
-			pm_runtime_enable(dev);
+			__pm_runtime_enable(dev, RPM_DISABLE_ENABLE_PM_SLEEP);
 		}
 		dev->power.direct_complete = false;
 	}
Index: linux-pm/drivers/base/power/runtime.c
===================================================================
--- linux-pm.orig/drivers/base/power/runtime.c
+++ linux-pm/drivers/base/power/runtime.c
@@ -353,6 +353,9 @@ static int rpm_idle(struct device *dev,
 
  out:
 	trace_rpm_return_int_rcuidle(dev, _THIS_IP_, retval);
+	if (retval && dev->power.pm_sleep_in_progress)
+		return 0;
+
 	return retval ? retval : rpm_suspend(dev, rpmflags | RPM_AUTO);
 }
 
@@ -550,6 +553,8 @@ static int rpm_suspend(struct device *de
 
  out:
 	trace_rpm_return_int(dev, _THIS_IP_, retval);
+	if (retval && dev->power.pm_sleep_in_progress)
+		return 0;
 
 	return retval;
 
@@ -765,6 +770,8 @@ static int rpm_resume(struct device *dev
 	}
 
 	trace_rpm_return_int_rcuidle(dev, _THIS_IP_, retval);
+	if (retval && dev->power.pm_sleep_in_progress)
+		return 0;
 
 	return retval;
 }
@@ -867,6 +874,8 @@ int pm_schedule_suspend(struct device *d
 
  out:
 	spin_unlock_irqrestore(&dev->power.lock, flags);
+	if (retval && dev->power.pm_sleep_in_progress)
+		return 0;
 
 	return retval;
 }
@@ -1200,28 +1209,35 @@ void __pm_runtime_disable(struct device
 		__pm_runtime_barrier(dev);
 
  out:
+	if (flags & RPM_DISABLE_ENABLE_PM_SLEEP)
+		dev->power.pm_sleep_in_progress = true;
+
 	spin_unlock_irq(&dev->power.lock);
 }
 EXPORT_SYMBOL_GPL(__pm_runtime_disable);
 
 /**
- * pm_runtime_enable - Enable runtime PM of a device.
+ * __pm_runtime_enable - Enable runtime PM of a device.
  * @dev: Device to handle.
+ * @flags: Behavior modifiers.
  */
-void pm_runtime_enable(struct device *dev)
+void __pm_runtime_enable(struct device *dev, unsigned int flags)
 {
-	unsigned long flags;
+	unsigned long irqflags;
 
-	spin_lock_irqsave(&dev->power.lock, flags);
+	spin_lock_irqsave(&dev->power.lock, irqflags);
+
+	if (flags & RPM_DISABLE_ENABLE_PM_SLEEP)
+		dev->power.pm_sleep_in_progress = false;
 
 	if (dev->power.disable_depth > 0)
 		dev->power.disable_depth--;
 	else
 		dev_warn(dev, "Unbalanced %s!\n", __func__);
 
-	spin_unlock_irqrestore(&dev->power.lock, flags);
+	spin_unlock_irqrestore(&dev->power.lock, irqflags);
 }
-EXPORT_SYMBOL_GPL(pm_runtime_enable);
+EXPORT_SYMBOL_GPL(__pm_runtime_enable);
 
 /**
  * pm_runtime_forbid - Block runtime PM of a device.
@@ -1396,6 +1412,7 @@ void pm_runtime_init(struct device *dev)
 	dev->power.idle_notification = false;
 
 	dev->power.disable_depth = 1;
+	dev->power.pm_sleep_in_progress = false;
 	atomic_set(&dev->power.usage_count, 0);
 
 	dev->power.runtime_error = 0;
Index: linux-pm/include/linux/pm.h
===================================================================
--- linux-pm.orig/include/linux/pm.h
+++ linux-pm/include/linux/pm.h
@@ -585,6 +585,7 @@ struct dev_pm_info {
 	atomic_t		usage_count;
 	atomic_t		child_count;
 	unsigned int		disable_depth:3;
+	bool			pm_sleep_in_progress:1;
 	unsigned int		idle_notification:1;
 	unsigned int		request_pending:1;
 	unsigned int		deferred_resume:1;
Index: linux-pm/include/linux/pm_runtime.h
===================================================================
--- linux-pm.orig/include/linux/pm_runtime.h
+++ linux-pm/include/linux/pm_runtime.h
@@ -25,6 +25,9 @@
 
 /* Runtime PM disable/enable flags */
 #define RPM_DISABLE_CHECK_RESUME	(1 << 0)
+#define RPM_DISABLE_ENABLE_PM_SLEEP	(1 << 1)
+
+#define RPM_DISABLE_ALL	(RPM_DISABLE_CHECK_RESUME | RPM_DISABLE_ENABLE_PM_SLEEP)
 
 #ifdef CONFIG_PM
 extern struct workqueue_struct *pm_wq;
@@ -46,7 +49,7 @@ extern int pm_runtime_get_if_in_use(stru
 extern int pm_schedule_suspend(struct device *dev, unsigned int delay);
 extern int __pm_runtime_set_status(struct device *dev, unsigned int status);
 extern int pm_runtime_barrier(struct device *dev);
-extern void pm_runtime_enable(struct device *dev);
+extern void __pm_runtime_enable(struct device *dev, unsigned int flags);
 extern void __pm_runtime_disable(struct device *dev, unsigned int flags);
 extern void pm_runtime_allow(struct device *dev);
 extern void pm_runtime_forbid(struct device *dev);
@@ -159,7 +162,7 @@ static inline int pm_runtime_get_if_in_u
 static inline int __pm_runtime_set_status(struct device *dev,
 					    unsigned int status) { return 0; }
 static inline int pm_runtime_barrier(struct device *dev) { return 0; }
-static inline void pm_runtime_enable(struct device *dev) {}
+static inline void __pm_runtime_enable(struct device *dev, unsigned int flags) {}
 static inline void __pm_runtime_disable(struct device *dev, unsigned int flags) {}
 static inline void pm_runtime_allow(struct device *dev) {}
 static inline void pm_runtime_forbid(struct device *dev) {}
@@ -278,6 +281,11 @@ static inline void pm_runtime_disable(st
 	__pm_runtime_disable(dev, RPM_DISABLE_CHECK_RESUME);
 }
 
+static inline void pm_runtime_enable(struct device *dev)
+{
+	__pm_runtime_enable(dev, 0);
+}
+
 static inline void pm_runtime_use_autosuspend(struct device *dev)
 {
 	__pm_runtime_use_autosuspend(dev, true);

Re: [RFC/RFT][PATCH v2 5/7] PM / runtime: Flag to indicate PM sleep transitions in progress

[Lukas Wunner]

On Thu, Sep 08, 2016 at 11:29:48PM +0200, Rafael J. Wysocki wrote:
> Introduce a new flag in struct dev_pm_info, pm_sleep_in_progress, to
> indicate that runtime PM has been disabled because of a PM sleep
> transition in progress.
[...]
> That will allow helpers like pm_runtime_get_sync() to be called
> during system sleep transitions without worrying about possible
> error codes they may return because runtime PM is disabled at
> that point.

I have a suspicion that this patch papers over the direct_complete bug
I reported Sep 10 and that the patch is unnecessary once that bug is
fixed.

AFAICS, runtime PM is only disabled in two places during the system
sleep process: In __device_suspend() for devices using direct_complete,
and __device_suspend_late() for all devices.

In both of these phases (dpm_suspend() and dpm_suspend_late()), the
device tree is walked bottom-up. Since we've reordered consumers to
the back of dpm_list, they will be treated *before* their suppliers.
Thus, runtime PM is disabled on the consumers first, and only later
on the suppliers.

Then how can it be that runtime PM is already disabled on the supplier?
The only scenario I can imagine is that the supplier chose to exercise
direct_complete, thus was pm_runtime_disabled() in the __device_suspend()
phase, and the consumer did *not* choose to exercise direct_complete and
later tried to runtime resume its suppliers and itself.

I assume this patch is a replacement for Marek's [v2 08/10].
@Marek, does this scenario match with what you witnessed?

Best regards,

Lukas

Re: [RFC/RFT][PATCH v2 5/7] PM / runtime: Flag to indicate PM sleep transitions in progress

[Rafael J. Wysocki]

On Monday, September 12, 2016 04:07:27 PM Lukas Wunner wrote:
> On Thu, Sep 08, 2016 at 11:29:48PM +0200, Rafael J. Wysocki wrote:
> > Introduce a new flag in struct dev_pm_info, pm_sleep_in_progress, to
> > indicate that runtime PM has been disabled because of a PM sleep
> > transition in progress.
> [...]
> > That will allow helpers like pm_runtime_get_sync() to be called
> > during system sleep transitions without worrying about possible
> > error codes they may return because runtime PM is disabled at
> > that point.
> 
> I have a suspicion that this patch papers over the direct_complete bug
> I reported Sep 10 and that the patch is unnecessary once that bug is
> fixed.

It doesn't paper over anything, but it may not be necessary anyway.

> AFAICS, runtime PM is only disabled in two places during the system
> sleep process: In __device_suspend() for devices using direct_complete,
> and __device_suspend_late() for all devices.
> 
> In both of these phases (dpm_suspend() and dpm_suspend_late()), the
> device tree is walked bottom-up. Since we've reordered consumers to
> the back of dpm_list, they will be treated *before* their suppliers.
> Thus, runtime PM is disabled on the consumers first, and only later
> on the suppliers.
> 
> Then how can it be that runtime PM is already disabled on the supplier?

Actually, I think that this was a consequence of a bug in device_reorder_to_tail()
that was present in the previous iteration of the patchset (it walked suppliers
instead of consumers).

> The only scenario I can imagine is that the supplier chose to exercise
> direct_complete, thus was pm_runtime_disabled() in the __device_suspend()
> phase, and the consumer did *not* choose to exercise direct_complete and
> later tried to runtime resume its suppliers and itself.
> 
> I assume this patch is a replacement for Marek's [v2 08/10].
> @Marek, does this scenario match with what you witnessed?

It is not strictly a replacement for it.  The Marek's patch was the
reason to post it, but I started to think about this earlier.

Some people have complained to me about having to deal with error codes
returned by the runtime PM framework during system suspend, so I thought
it might be useful to deal with that too.

That said it probably is not necessary right now.

Thanks,
Rafael

Re: [RFC/RFT][PATCH v2 5/7] PM / runtime: Flag to indicate PM sleep transitions in progress

[Lukas Wunner]

On Mon, Sep 12, 2016 at 11:25:36PM +0200, Rafael J. Wysocki wrote:
> On Monday, September 12, 2016 04:07:27 PM Lukas Wunner wrote:
> > On Thu, Sep 08, 2016 at 11:29:48PM +0200, Rafael J. Wysocki wrote:
> > > Introduce a new flag in struct dev_pm_info, pm_sleep_in_progress, to
> > > indicate that runtime PM has been disabled because of a PM sleep
> > > transition in progress.
> > [...]
> > > That will allow helpers like pm_runtime_get_sync() to be called
> > > during system sleep transitions without worrying about possible
> > > error codes they may return because runtime PM is disabled at
> > > that point.
> > 
> > I have a suspicion that this patch papers over the direct_complete bug
> > I reported Sep 10 and that the patch is unnecessary once that bug is
> > fixed.
> 
> It doesn't paper over anything, but it may not be necessary anyway.
> 
> > AFAICS, runtime PM is only disabled in two places during the system
> > sleep process: In __device_suspend() for devices using direct_complete,
> > and __device_suspend_late() for all devices.
> > 
> > In both of these phases (dpm_suspend() and dpm_suspend_late()), the
> > device tree is walked bottom-up. Since we've reordered consumers to
> > the back of dpm_list, they will be treated *before* their suppliers.
> > Thus, runtime PM is disabled on the consumers first, and only later
> > on the suppliers.
> > 
> > Then how can it be that runtime PM is already disabled on the supplier?
> 
> Actually, I think that this was a consequence of a bug in
> device_reorder_to_tail() that was present in the previous iteration
> of the patchset (it walked suppliers instead of consumers).
> 
> > The only scenario I can imagine is that the supplier chose to exercise
> > direct_complete, thus was pm_runtime_disabled() in the __device_suspend()
> > phase, and the consumer did *not* choose to exercise direct_complete and
> > later tried to runtime resume its suppliers and itself.
> > 
> > I assume this patch is a replacement for Marek's [v2 08/10].
> > @Marek, does this scenario match with what you witnessed?
> 
> It is not strictly a replacement for it.  The Marek's patch was the
> reason to post it, but I started to think about this earlier.
> 
> Some people have complained to me about having to deal with error codes
> returned by the runtime PM framework during system suspend, so I thought
> it might be useful to deal with that too.
> 
> That said it probably is not necessary right now.

Understood, thanks for providing this context which was unknown to me.

I'm wondering if it's necessary to introduce a new "pm_sleep_in_progress"
flag. We've already got "is_prepared", "is_suspended", "is_late_suspended",
"is_noirq_suspended", so we should have a pretty good idea of the fact
that the device is going to sleep and which stage it's in.

E.g. (dev->power.direct_complete || dev->power.is_suspended) covers a bit
more than the time frame when runtime PM is disabled for system sleep,
but might perhaps still suffice as a proxy.

Should a new flag be unavoidable, setting it directly in
__device_suspend_late(), device_resume_early(), __device_suspend()
and device_resume() would result in a smaller patch. (E.g. you
wouldn't have to modify the prototype of pm_runtime_enable().)

Thanks,

Lukas

Re: [RFC/RFT][PATCH v2 5/7] PM / runtime: Flag to indicate PM sleep transitions in progress

[Marek Szyprowski]

Hi Rafael,


On 2016-09-12 23:25, Rafael J. Wysocki wrote:
> On Monday, September 12, 2016 04:07:27 PM Lukas Wunner wrote:
>> On Thu, Sep 08, 2016 at 11:29:48PM +0200, Rafael J. Wysocki wrote:
>>> Introduce a new flag in struct dev_pm_info, pm_sleep_in_progress, to
>>> indicate that runtime PM has been disabled because of a PM sleep
>>> transition in progress.
>> [...]
>>> That will allow helpers like pm_runtime_get_sync() to be called
>>> during system sleep transitions without worrying about possible
>>> error codes they may return because runtime PM is disabled at
>>> that point.
>> I have a suspicion that this patch papers over the direct_complete bug
>> I reported Sep 10 and that the patch is unnecessary once that bug is
>> fixed.
> It doesn't paper over anything, but it may not be necessary anyway.
>
>> AFAICS, runtime PM is only disabled in two places during the system
>> sleep process: In __device_suspend() for devices using direct_complete,
>> and __device_suspend_late() for all devices.
>>
>> In both of these phases (dpm_suspend() and dpm_suspend_late()), the
>> device tree is walked bottom-up. Since we've reordered consumers to
>> the back of dpm_list, they will be treated *before* their suppliers.
>> Thus, runtime PM is disabled on the consumers first, and only later
>> on the suppliers.
>>
>> Then how can it be that runtime PM is already disabled on the supplier?
> Actually, I think that this was a consequence of a bug in device_reorder_to_tail()
> that was present in the previous iteration of the patchset (it walked suppliers
> instead of consumers).
>
>> The only scenario I can imagine is that the supplier chose to exercise
>> direct_complete, thus was pm_runtime_disabled() in the __device_suspend()
>> phase, and the consumer did *not* choose to exercise direct_complete and
>> later tried to runtime resume its suppliers and itself.
>>
>> I assume this patch is a replacement for Marek's [v2 08/10].
>> @Marek, does this scenario match with what you witnessed?
> It is not strictly a replacement for it.  The Marek's patch was the
> reason to post it, but I started to think about this earlier.
>
> Some people have complained to me about having to deal with error codes
> returned by the runtime PM framework during system suspend, so I thought
> it might be useful to deal with that too.
>
> That said it probably is not necessary right now.

I've tested this patchset without this patch and system sleep with 
device link
enabled worked fine. However this might be also a consequence of 
enabling runtime
pm during system sleep since v4.8-rc1.

It looks that for now this patch can be skipped until a real use case for it
appears.

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [RFC/RFT][PATCH v2 5/7] PM / runtime: Flag to indicate PM sleep transitions in progress

[Rafael J. Wysocki]

On Tuesday, September 13, 2016 09:21:23 AM Marek Szyprowski wrote:
> Hi Rafael,
> 
> 
> On 2016-09-12 23:25, Rafael J. Wysocki wrote:
> > On Monday, September 12, 2016 04:07:27 PM Lukas Wunner wrote:
> >> On Thu, Sep 08, 2016 at 11:29:48PM +0200, Rafael J. Wysocki wrote:
> >>> Introduce a new flag in struct dev_pm_info, pm_sleep_in_progress, to
> >>> indicate that runtime PM has been disabled because of a PM sleep
> >>> transition in progress.
> >> [...]
> >>> That will allow helpers like pm_runtime_get_sync() to be called
> >>> during system sleep transitions without worrying about possible
> >>> error codes they may return because runtime PM is disabled at
> >>> that point.
> >> I have a suspicion that this patch papers over the direct_complete bug
> >> I reported Sep 10 and that the patch is unnecessary once that bug is
> >> fixed.
> > It doesn't paper over anything, but it may not be necessary anyway.
> >
> >> AFAICS, runtime PM is only disabled in two places during the system
> >> sleep process: In __device_suspend() for devices using direct_complete,
> >> and __device_suspend_late() for all devices.
> >>
> >> In both of these phases (dpm_suspend() and dpm_suspend_late()), the
> >> device tree is walked bottom-up. Since we've reordered consumers to
> >> the back of dpm_list, they will be treated *before* their suppliers.
> >> Thus, runtime PM is disabled on the consumers first, and only later
> >> on the suppliers.
> >>
> >> Then how can it be that runtime PM is already disabled on the supplier?
> > Actually, I think that this was a consequence of a bug in device_reorder_to_tail()
> > that was present in the previous iteration of the patchset (it walked suppliers
> > instead of consumers).
> >
> >> The only scenario I can imagine is that the supplier chose to exercise
> >> direct_complete, thus was pm_runtime_disabled() in the __device_suspend()
> >> phase, and the consumer did *not* choose to exercise direct_complete and
> >> later tried to runtime resume its suppliers and itself.
> >>
> >> I assume this patch is a replacement for Marek's [v2 08/10].
> >> @Marek, does this scenario match with what you witnessed?
> > It is not strictly a replacement for it.  The Marek's patch was the
> > reason to post it, but I started to think about this earlier.
> >
> > Some people have complained to me about having to deal with error codes
> > returned by the runtime PM framework during system suspend, so I thought
> > it might be useful to deal with that too.
> >
> > That said it probably is not necessary right now.
> 
> I've tested this patchset without this patch and system sleep with 
> device link
> enabled worked fine. However this might be also a consequence of 
> enabling runtime
> pm during system sleep since v4.8-rc1.
> 
> It looks that for now this patch can be skipped until a real use case for it
> appears.

OK, thanks!

[RFC/RFT][PATCH v2 6/7] PM / runtime: Use device links

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Modify the runtime PM framework to use device links to ensure that
supplier devices will not be suspended if any of their consumer
devices are active.

The idea is to reference count suppliers on the consumer's resume
and drop references to them on its suspend.  The information on
whether or not the supplier has been reference counted by the
consumer's (runtime) resume is stored in a new field (rpm_active)
in the link object for each link.

It may be necessary to clean up those references when the
supplier is unbinding and that's why the links whose status is
DEVICE_LINK_SUPPLIER_UNBIND are skipped by the runtime suspend
and resume code.

The above means that if the consumer device is probed in the
runtime-active state, the supplier has to be resumed and reference
counted by device_link_add() so the code works as expected on its
(runtime) suspend.  There is a new flag, DEVICE_LINK_RPM_ACTIVE,
to tell device_link_add() about that (in which case the caller
is responsible for making sure that the consumer really will
be runtime-active when runtime PM is enabled for it).

The other new link flag, DEVICE_LINK_PM_RUNTIME, tells the core
whether or not the link should be used for runtime PM at all.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/base/core.c          |   15 ++++++
 drivers/base/dd.c            |    1 
 drivers/base/power/runtime.c |   93 ++++++++++++++++++++++++++++++++++++++++---
 include/linux/device.h       |    5 ++
 include/linux/pm_runtime.h   |    2 
 5 files changed, 111 insertions(+), 5 deletions(-)

Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -110,6 +110,11 @@ struct device_link *device_link_add(stru
 	if (!consumer || !supplier || status == DEVICE_LINK_SUPPLIER_UNBIND)
 		return NULL;
 
+	if ((flags & DEVICE_LINK_RPM_ACTIVE) &&
+	    (!(flags & DEVICE_LINK_PM_RUNTIME) ||
+	     status != DEVICE_LINK_CONSUMER_PROBE))
+		return NULL;
+
 	mutex_lock(&device_links_lock);
 
 	/*
@@ -129,6 +134,16 @@ struct device_link *device_link_add(stru
 	if (!link)
 		goto out;
 
+	if (flags & DEVICE_LINK_RPM_ACTIVE) {
+		if (pm_runtime_get_sync(supplier) < 0) {
+			pm_runtime_put_noidle(supplier);
+			kfree(link);
+			goto out;
+		}
+		link->rpm_active = true;
+	} else {
+		link->rpm_active = false;
+	}
 	get_device(supplier);
 	link->supplier = supplier;
 	INIT_LIST_HEAD(&link->s_node);
Index: linux-pm/drivers/base/dd.c
===================================================================
--- linux-pm.orig/drivers/base/dd.c
+++ linux-pm/drivers/base/dd.c
@@ -791,6 +791,7 @@ static void __device_release_driver(stru
 		}
 
 		pm_runtime_get_sync(dev);
+		pm_runtime_clean_up_links(dev);
 
 		driver_sysfs_remove(dev);
 
Index: linux-pm/drivers/base/power/runtime.c
===================================================================
--- linux-pm.orig/drivers/base/power/runtime.c
+++ linux-pm/drivers/base/power/runtime.c
@@ -12,6 +12,8 @@
 #include <linux/pm_runtime.h>
 #include <linux/pm_wakeirq.h>
 #include <trace/events/rpm.h>
+
+#include "../base.h"
 #include "power.h"
 
 typedef int (*pm_callback_t)(struct device *);
@@ -266,19 +268,69 @@ static int rpm_check_suspend_allowed(str
 static int __rpm_callback(int (*cb)(struct device *), struct device *dev)
 	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
-	int retval;
+	struct device_link *link;
+	int retval, idx;
 
-	if (dev->power.irq_safe)
+	if (dev->power.irq_safe) {
 		spin_unlock(&dev->power.lock);
-	else
+	} else {
 		spin_unlock_irq(&dev->power.lock);
 
+		/*
+		 * Resume suppliers if necessary.
+		 *
+		 * The device's runtime PM status cannot change until this
+		 * routine returns, so it is safe to read the status outside of
+		 * the lock.
+		 */
+		if (dev->power.runtime_status == RPM_RESUMING) {
+			idx = device_links_read_lock();
+
+			list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node)
+				if ((link->flags & DEVICE_LINK_PM_RUNTIME)
+				    && link->status != DEVICE_LINK_SUPPLIER_UNBIND
+				    && !link->rpm_active) {
+					retval = pm_runtime_get_sync(link->supplier);
+					if (retval < 0) {
+						pm_runtime_put_noidle(link->supplier);
+						goto fail;
+					}
+					link->rpm_active = true;
+				}
+
+			device_links_read_unlock(idx);
+		}
+	}
+
 	retval = cb(dev);
 
-	if (dev->power.irq_safe)
+	if (dev->power.irq_safe) {
 		spin_lock(&dev->power.lock);
-	else
+	} else {
+		/*
+		 * If the device is suspending and the callback has returned
+		 * success, drop the usage counters of the suppliers that have
+		 * been reference counted on its resume.
+		 *
+		 * Do that if resume fails too.
+		 */
+		if ((dev->power.runtime_status == RPM_SUSPENDING && !retval)
+		    || (dev->power.runtime_status == RPM_RESUMING && retval)) {
+			idx = device_links_read_lock();
+
+ fail:
+			list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node)
+				if (link->status != DEVICE_LINK_SUPPLIER_UNBIND
+				    && link->rpm_active) {
+					pm_runtime_put(link->supplier);
+					link->rpm_active = false;
+				}
+
+			device_links_read_unlock(idx);
+		}
+
 		spin_lock_irq(&dev->power.lock);
+	}
 
 	return retval;
 }
@@ -1464,6 +1516,37 @@ void pm_runtime_remove(struct device *de
 }
 
 /**
+ * pm_runtime_clean_up_links - Prepare links to consumers for driver removal.
+ * @dev: Device whose driver is going to be removed.
+ *
+ * Check links from this device to any consumers and if any of them have active
+ * runtime PM references to the device, drop the usage counter of the device
+ * (once per link).
+ *
+ * Since the device is guaranteed to be runtime-active at the point this is
+ * called, nothing else needs to be done here.
+ *
+ * Moreover, this is called after device_links_busy() has returned 'false', so
+ * the status of each link is guaranteed to be DEVICE_LINK_SUPPLIER_UNBIND and
+ * therefore rpm_active can't be manipulated concurrently.
+ */
+void pm_runtime_clean_up_links(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node)
+		if (link->rpm_active) {
+			pm_runtime_put_noidle(dev);
+			link->rpm_active = false;
+		}
+
+	device_links_read_unlock(idx);
+}
+
+/**
  * pm_runtime_force_suspend - Force a device into suspend state if needed.
  * @dev: Device to suspend.
  *
Index: linux-pm/include/linux/device.h
===================================================================
--- linux-pm.orig/include/linux/device.h
+++ linux-pm/include/linux/device.h
@@ -718,8 +718,12 @@ enum device_link_status {
  * Device link flags.
  *
  * PERSISTENT: Do not delete the link on consumer device driver unbind.
+ * PM_RUNTIME: If set, the runtime PM framework will use this link.
+ * RPM_ACTIVE: Run pm_runtime_get_sync() on the supplier during link creation.
  */
 #define DEVICE_LINK_PERSISTENT	(1 << 0)
+#define DEVICE_LINK_PM_RUNTIME	(1 << 1)
+#define DEVICE_LINK_RPM_ACTIVE	(1 << 2)
 
 struct device_link {
 	struct device *supplier;
@@ -728,6 +732,7 @@ struct device_link {
 	struct list_head c_node;
 	enum device_link_status status;
 	u32 flags;
+	bool rpm_active;
 	spinlock_t lock;
 	struct rcu_head rcu_head;
 };
Index: linux-pm/include/linux/pm_runtime.h
===================================================================
--- linux-pm.orig/include/linux/pm_runtime.h
+++ linux-pm/include/linux/pm_runtime.h
@@ -61,6 +61,7 @@ extern unsigned long pm_runtime_autosusp
 extern void pm_runtime_update_max_time_suspended(struct device *dev,
 						 s64 delta_ns);
 extern void pm_runtime_set_memalloc_noio(struct device *dev, bool enable);
+extern void pm_runtime_clean_up_links(struct device *dev);
 
 static inline void pm_suspend_ignore_children(struct device *dev, bool enable)
 {
@@ -192,6 +193,7 @@ static inline unsigned long pm_runtime_a
 				struct device *dev) { return 0; }
 static inline void pm_runtime_set_memalloc_noio(struct device *dev,
 						bool enable){}
+static inline void pm_runtime_clean_up_links(struct device *dev) {}
 
 #endif /* !CONFIG_PM */
 

Re: [RFC/RFT][PATCH v2 6/7] PM / runtime: Use device links

[Lukas Wunner]

On Thu, Sep 08, 2016 at 11:30:26PM +0200, Rafael J. Wysocki wrote:
> Modify the runtime PM framework to use device links to ensure that
> supplier devices will not be suspended if any of their consumer
> devices are active.

I think it's inconsistent to runtime resume/suspend suppliers in
__rpm_callback() whereas the parent is treated in rpm_suspend()
and rpm_resume().

Instead I'd suggest to amend struct dev_pm_ops with:
	atomic_t		consumer_count;

Amend rpm_check_suspend_allowed() with:
	else if (atomic_read(&dev->power.consumer_count) > 0)
		retval = -EBUSY;

Amend rpm_suspend(), rpm_resume() and __pm_runtime_set_status()
to decrement/increment consumer_count where we're doing the same
for the parent's child_count, and runtime resume/idle suppliers
as necessary.


> The idea is to reference count suppliers on the consumer's resume
> and drop references to them on its suspend.  The information on
> whether or not the supplier has been reference counted by the
> consumer's (runtime) resume is stored in a new field (rpm_active)
> in the link object for each link.

So the rpm_active variable indicates if the runtime ref on the
supplier is currently held. I don't see why this is needed.
If DEVICE_LINK_PM_RUNTIME is not set, we never acquire a runtime
ref in the first place. If it's set, a ref is acquired upon
resuming the consumer and released upon suspending it. So whether
the ref is held is discernable from the consumer's runtime PM state.
Why do you need to track this in a separate variable?


> @@ -718,8 +718,12 @@ enum device_link_status {
>   * Device link flags.
>   *
>   * PERSISTENT: Do not delete the link on consumer device driver unbind.
> + * PM_RUNTIME: If set, the runtime PM framework will use this link.
> + * RPM_ACTIVE: Run pm_runtime_get_sync() on the supplier during link creation.
>   */
>  #define DEVICE_LINK_PERSISTENT	(1 << 0)
> +#define DEVICE_LINK_PM_RUNTIME	(1 << 1)
> +#define DEVICE_LINK_RPM_ACTIVE	(1 << 2)

I don't understand the need for DEVICE_LINK_RPM_ACTIVE: If the
consumer is in runtime resumed state when the link is added and
DEVICE_LINK_PM_RUNTIME is set, then of course the supplier needs
to be in runtime resumed state as well. Conversely if the consumer
is in runtime suspended state, the supplier need not be in runtime
resumed state either. So the value of the flag can be derived from
the consumer's runtime PM state. Why do we need the flag at all?

Thanks,

Lukas

Re: [RFC/RFT][PATCH v2 6/7] PM / runtime: Use device links

[Rafael J. Wysocki]

On Monday, September 12, 2016 11:47:58 AM Lukas Wunner wrote:
> On Thu, Sep 08, 2016 at 11:30:26PM +0200, Rafael J. Wysocki wrote:
> > Modify the runtime PM framework to use device links to ensure that
> > supplier devices will not be suspended if any of their consumer
> > devices are active.
> 
> I think it's inconsistent to runtime resume/suspend suppliers in
> __rpm_callback() whereas the parent is treated in rpm_suspend()
> and rpm_resume().

The reason why I did that this way is the rollback needed in case of
errors and that led to duplicated code if done elsewhere.

> Instead I'd suggest to amend struct dev_pm_ops with:
> 	atomic_t		consumer_count;
> 
> Amend rpm_check_suspend_allowed() with:
> 	else if (atomic_read(&dev->power.consumer_count) > 0)
> 		retval = -EBUSY;

That is a good idea, though (from the first look at least).

> Amend rpm_suspend(), rpm_resume() and __pm_runtime_set_status()
> to decrement/increment consumer_count where we're doing the same
> for the parent's child_count, and runtime resume/idle suppliers
> as necessary.
> 
> 
> > The idea is to reference count suppliers on the consumer's resume
> > and drop references to them on its suspend.  The information on
> > whether or not the supplier has been reference counted by the
> > consumer's (runtime) resume is stored in a new field (rpm_active)
> > in the link object for each link.
> 
> So the rpm_active variable indicates if the runtime ref on the
> supplier is currently held. I don't see why this is needed.
> If DEVICE_LINK_PM_RUNTIME is not set, we never acquire a runtime
> ref in the first place. If it's set, a ref is acquired upon
> resuming the consumer and released upon suspending it. So whether
> the ref is held is discernable from the consumer's runtime PM state.
> Why do you need to track this in a separate variable?

Please see pm_runtime_clean_up_links().

> > @@ -718,8 +718,12 @@ enum device_link_status {
> >   * Device link flags.
> >   *
> >   * PERSISTENT: Do not delete the link on consumer device driver unbind.
> > + * PM_RUNTIME: If set, the runtime PM framework will use this link.
> > + * RPM_ACTIVE: Run pm_runtime_get_sync() on the supplier during link creation.
> >   */
> >  #define DEVICE_LINK_PERSISTENT	(1 << 0)
> > +#define DEVICE_LINK_PM_RUNTIME	(1 << 1)
> > +#define DEVICE_LINK_RPM_ACTIVE	(1 << 2)
> 
> I don't understand the need for DEVICE_LINK_RPM_ACTIVE: If the
> consumer is in runtime resumed state when the link is added and
> DEVICE_LINK_PM_RUNTIME is set, then of course the supplier needs
> to be in runtime resumed state as well. Conversely if the consumer
> is in runtime suspended state, the supplier need not be in runtime
> resumed state either. So the value of the flag can be derived from
> the consumer's runtime PM state. Why do we need the flag at all?

That's because device_link_add() doesn't know that runtime PM states of
the supplier/consumer and the flag tells it what to do (with the assumption
that the caller knows the situation, of course).

Thanks,
Rafael

Re: [RFC/RFT][PATCH v2 6/7] PM / runtime: Use device links

[Rafael J. Wysocki]

On Monday, September 12, 2016 03:57:02 PM Rafael J. Wysocki wrote:
> On Monday, September 12, 2016 11:47:58 AM Lukas Wunner wrote:
> > On Thu, Sep 08, 2016 at 11:30:26PM +0200, Rafael J. Wysocki wrote:
> > > Modify the runtime PM framework to use device links to ensure that
> > > supplier devices will not be suspended if any of their consumer
> > > devices are active.
> > 
> > I think it's inconsistent to runtime resume/suspend suppliers in
> > __rpm_callback() whereas the parent is treated in rpm_suspend()
> > and rpm_resume().
> 
> The reason why I did that this way is the rollback needed in case of
> errors and that led to duplicated code if done elsewhere.
> 
> > Instead I'd suggest to amend struct dev_pm_ops with:
> > 	atomic_t		consumer_count;
> > 
> > Amend rpm_check_suspend_allowed() with:
> > 	else if (atomic_read(&dev->power.consumer_count) > 0)
> > 		retval = -EBUSY;
> 
> That is a good idea, though (from the first look at least).

No, I actually don't like it.

Suppliers are not parents and even the separate counter we have for
parents is somewhat artificial.

For parents it may be argued that they are always there, so handling them
in a special way is justified, but for suppliers I really don't see why
adding a new counter would be better.

Thanks,
Rafael

[RFC/RFT][PATCH v2 7/7] PM / runtime: Optimize the use of device links

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

If the device has no links to suppliers that should be used for
runtime PM (links with DEVICE_LINK_PM_RUNTIME set), there is no
reason to walk the list of suppliers for that device during
runtime suspend and resume.

Add a simple mechanism to detect that case and possibly avoid the
extra unnecessary overhead.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/base/core.c          |    6 ++++++
 drivers/base/power/runtime.c |   23 ++++++++++++++++++++---
 include/linux/pm.h           |    1 +
 include/linux/pm_runtime.h   |    4 ++++
 4 files changed, 31 insertions(+), 3 deletions(-)

Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -151,6 +151,9 @@ struct device_link *device_link_add(stru
 	link->consumer = consumer;
 	INIT_LIST_HEAD(&link->c_node);
 	spin_lock_init(&link->lock);
+	if (flags & DEVICE_LINK_PM_RUNTIME)
+		pm_runtime_new_link(consumer);
+
 	link->flags = flags;
 	link->status = status;
 
@@ -191,6 +194,9 @@ static void __device_link_del(struct dev
 	dev_info(link->consumer, "Dropping the link to %s\n",
 		 dev_name(link->supplier));
 
+	if (link->flags & DEVICE_LINK_PM_RUNTIME)
+		pm_runtime_drop_link(link->consumer);
+
 	list_del_rcu(&link->s_node);
 	list_del_rcu(&link->c_node);
 	call_srcu(&device_links_srcu, &link->rcu_head, __device_link_free_srcu);
Index: linux-pm/drivers/base/power/runtime.c
===================================================================
--- linux-pm.orig/drivers/base/power/runtime.c
+++ linux-pm/drivers/base/power/runtime.c
@@ -270,6 +270,7 @@ static int __rpm_callback(int (*cb)(stru
 {
 	struct device_link *link;
 	int retval, idx;
+	bool use_links = dev->power.links_count > 0;
 
 	if (dev->power.irq_safe) {
 		spin_unlock(&dev->power.lock);
@@ -283,7 +284,7 @@ static int __rpm_callback(int (*cb)(stru
 		 * routine returns, so it is safe to read the status outside of
 		 * the lock.
 		 */
-		if (dev->power.runtime_status == RPM_RESUMING) {
+		if (use_links && dev->power.runtime_status == RPM_RESUMING) {
 			idx = device_links_read_lock();
 
 			list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node)
@@ -314,8 +315,9 @@ static int __rpm_callback(int (*cb)(stru
 		 *
 		 * Do that if resume fails too.
 		 */
-		if ((dev->power.runtime_status == RPM_SUSPENDING && !retval)
-		    || (dev->power.runtime_status == RPM_RESUMING && retval)) {
+		if (use_links
+		    && ((dev->power.runtime_status == RPM_SUSPENDING && !retval)
+		    || (dev->power.runtime_status == RPM_RESUMING && retval))) {
 			idx = device_links_read_lock();
 
  fail:
@@ -1546,6 +1548,21 @@ void pm_runtime_clean_up_links(struct de
 	device_links_read_unlock(idx);
 }
 
+void pm_runtime_new_link(struct device *dev)
+{
+	spin_lock_irq(&dev->power.lock);
+	dev->power.links_count++;
+	spin_unlock_irq(&dev->power.lock);
+}
+
+void pm_runtime_drop_link(struct device *dev)
+{
+	spin_lock_irq(&dev->power.lock);
+	WARN_ON(dev->power.links_count == 0);
+	dev->power.links_count--;
+	spin_unlock_irq(&dev->power.lock);
+}
+
 /**
  * pm_runtime_force_suspend - Force a device into suspend state if needed.
  * @dev: Device to suspend.
Index: linux-pm/include/linux/pm.h
===================================================================
--- linux-pm.orig/include/linux/pm.h
+++ linux-pm/include/linux/pm.h
@@ -597,6 +597,7 @@ struct dev_pm_info {
 	unsigned int		use_autosuspend:1;
 	unsigned int		timer_autosuspends:1;
 	unsigned int		memalloc_noio:1;
+	unsigned int		links_count;
 	enum rpm_request	request;
 	enum rpm_status		runtime_status;
 	int			runtime_error;
Index: linux-pm/include/linux/pm_runtime.h
===================================================================
--- linux-pm.orig/include/linux/pm_runtime.h
+++ linux-pm/include/linux/pm_runtime.h
@@ -62,6 +62,8 @@ extern void pm_runtime_update_max_time_s
 						 s64 delta_ns);
 extern void pm_runtime_set_memalloc_noio(struct device *dev, bool enable);
 extern void pm_runtime_clean_up_links(struct device *dev);
+extern void pm_runtime_new_link(struct device *dev);
+extern void pm_runtime_drop_link(struct device *dev);
 
 static inline void pm_suspend_ignore_children(struct device *dev, bool enable)
 {
@@ -194,6 +196,8 @@ static inline unsigned long pm_runtime_a
 static inline void pm_runtime_set_memalloc_noio(struct device *dev,
 						bool enable){}
 static inline void pm_runtime_clean_up_links(struct device *dev) {}
+static inline void pm_runtime_new_link(struct device *dev) {}
+static inline void pm_runtime_drop_link(struct device *dev) {}
 
 #endif /* !CONFIG_PM */
 

Re: [RFC/RFT][PATCH v2 0/7] Functional dependencies between devices

[Rafael J. Wysocki]

On Thursday, September 08, 2016 11:25:44 PM Rafael J. Wysocki wrote:
> Hi Everyone,
> 
> This is a refresh of the functional dependencies series that I posted last
> year and which has picked up by Marek quite recently.  For reference, appended
> is my introductory message sent previously (which may be slightly outdated now).
> 
> As last time, the first patch rearranges the code around __device_release_driver()
> a bit to prepare it for the next one (it actually hasn't changed AFAICS).
> 
> The second patch introduces the actual device links mechanics, but without
> system suspend/resume and runtime PM support which are added by the subsequent
> patches.
> 
> Some bugs found by Marek during his work on these patches should be fixed
> here.  In particular, the endless recursion in device_reorder_to_tail()
> which simply was broken before.
> 
> There are two additional patches to address the issue with runtime PM support
> that occured when runtime PM was disabled for some suppliers due to a PM
> sleep transition in progress.  Those patches simply make runtime PM helpers
> return 0 in that case which may be controversial, so please let me know if
> there are concerns about those.
> 
> The way device_link_add() works is a bit different, as it takes an additional
> status argument now.  That makes it possible to create a link in any state,
> with extra care of course, and should address the problem pointed to by Lukas
> during the previous discussion.
> 
> Also some comments from Tomeu have been addressed.
> 
> This hasn't been really tested yet and I'm sort of relying on Marek to test
> it, because he has a use case ready.  Hence, the RFT tag on the series.
> 
> Overall, please let me know what you think.
> 
> Thanks,
> Rafael
> 
> 
> Introduction:
> 
> As discussed in the recent "On-demand device probing" thread and in a Kernel
> Summit session earlier today, there is a problem with handling cases where
> functional dependencies between devices are involved.
> 
> What I mean by a "functional dependency" is when the driver of device B needs
> both device A and its driver to be present and functional to be able to work.
> This implies that the driver of A needs to be working for B to be probed
> successfully and it cannot be unbound from the device before the B's driver.
> This also has certain consequences for power management of these devices
> (suspend/resume and runtime PM ordering).
> 
> So I want to be able to represent those functional dependencies between devices
> and I'd like the driver core to track them and act on them in certain cases
> where they matter.  The argument for doing that in the driver core is that
> there are quite a few distinct use cases related to that, they are relatively
> hard to get right in a driver (if one wants to address all of them properly)
> and it only gets worse if multiplied by the number of drivers potentially
> needing to do it.  Morever, at least one case (asynchronous system suspend/resume)
> cannot be handled in a single driver at all, because it requires the driver of A
> to wait for B to suspend (during system suspend) and the driver of B to wait for
> A to resume (during system resume).
> 
> My idea is to represent a supplier-consumer dependency between devices (or
> more precisely between device+driver combos) as a "link" object containing
> pointers to the devices in question, a list node for each of them and some
> additional information related to the management of those objects, ie.
> something like:
> 
> struct device_link {
>         struct device *supplier;
>         struct list_head supplier_node;
>         struct device *consumer;
>         struct list_head consumer_node;
>         <flags, status etc>
> };
> 
> In general, there will be two lists of those things per device, one list
> of links to consumers and one list of links to suppliers.
> 
> In that picture, links will be created by calling, say:
> 
> int device_add_link(struct device *me, struct device *my_supplier, unsigned int flags);
> 
> and they will be deleted by the driver core when not needed any more.  The
> creation of a link should also cause dpm_list and the list used during shutdown
> to be reordered if needed.
> 
> In principle, it seems usefult to consider two types of links, one created
> at device registration time (when registering the second device from the linked
> pair, whichever it is) and one created at probe time (of the consumer device).
> I'll refer to them as "permanent" and "probe-time" links, respectively.
> 
> The permanent links (created at device registration time) will stay around
> until one of the linked devices is unregistered (at which time the driver
> core will drop the link along with the device going away).  The probe-time
> ones will be dropped (automatically) at the consumer device driver unbind time.
> 
> There's a question about what if the supplier device is being unbound before
> the consumer one (for example, as a result of a hotplug event).  My current
> view on that is that the consumer needs to be force-unbound in that case too,
> but I guess I may be persuaded otherwise given sufficiently convincing
> arguments.  Anyway, there are reasons to do that, like for example it may
> help with the synchronization.  Namely, if there's a rule that suppliers
> cannot be unbound before any consumers linked to them, than the list of links
> to suppliers for a consumer can only change at its registration/probe or
> unbind/remove times (which simplifies things quite a bit).
> 
> With that, the permanent links existing at the probe time for a consumer
> device can be used to check whether or not to defer the probing of it
> even before executing its probe callback.  In turn, system suspend
> synchronization should be a matter of calling device_pm_wait_for_dev()
> for all consumers of a supplier device, in analogy with dpm_wait_for_children(),
> and so on.
> 
> Of course, the new lists have to be stable during those operations and ensuring
> that is going to be somewhat tricky (AFAICS right now at least), but apart from
> that the whole concept looks reasonably straightforward to me.
> --

The Mark's address is broken in this series.  Again, sadly.

Really sorry about that and please fix it up when you reply.

Thanks,
Rafael

Re: [RFC/RFT][PATCH v2 0/7] Functional dependencies between devices

[Lukas Wunner]

On Thu, Sep 08, 2016 at 11:25:44PM +0200, Rafael J. Wysocki wrote:
> As discussed in the recent "On-demand device probing" thread and in a
> Kernel Summit session earlier today, there is a problem with handling
> cases where functional dependencies between devices are involved.
> 
> What I mean by a "functional dependency" is when the driver of device B
> needs both device A and its driver to be present and functional to be
> able to work.  This implies that the driver of A needs to be working
> for B to be probed successfully and it cannot be unbound from the
> device before the B's driver.  This also has certain consequences for
> power management of these devices (suspend/resume and runtime PM
> ordering).

As a general observation, this series seems to conflate two
separate issues:

(a) non-hierarchical device dependencies
    (a device depends on another device which is not its parent
    with regards to (runtime) suspend/resume ordering)

(b) driver-presence dependencies
    (a device depends on another device to be bound to a driver
    before it can probe / must be unbound before the other device
    can unbind)

Those two issues are orthogonal.

E.g. a driver-presence dependency may exist between a child and a
parent, or between siblings, whereas a non-hierarchical device
dependency by definition cannot exist between child/parent.

Let's say I need a driver-presence dependency between parent/child.
The PM core already guarantees correct (runtime) suspend/resume
ordering between child. Device links duplicate that functionality.
Oops?

In a way, the approach taken here is somewhat contrary to the UNIX
philosophy to write programs that do one thing and do that well.
It's a single tool which addresses two distinct problems and I think
that makes it more difficult to understand the consequences of
device links, in particular if used between parent/child, and ensure
correctness.

Would it be worth to address issue (b) driver-presence dependencies
with a separate tool? I could envision adding two bits to struct device,
one to indicate that the device must be bound before its children/
consumers can bind, and another to indicate that the parent/suppliers
must be bound before the device can bind. The PM core could check those
bits to decide if it should defer probing. All the code for maintaining
device link state would probably no longer be necessary.

That approach would be more coarse-grained than setting up links
between two devices, but I imagine it would probably suffice in the
majority of use cases. If drivers need something more fine-grained,
they can always test boundness of suppliers in their ->probe hook
and return -EPROBE_DEFER. And they could handle unbinding before
suppliers by listening to BUS_NOTIFY_UNBIND_DRIVER.

As for (a) non-hierarchical device dependencies, that's a gaping
hole that we currently have. Off the cuff I can name 4 such device
dependencies on MacBooks alone where we currently try to make do
with various kludges to ensure correct suspend/resume ordering.
Being able to express such dependencies in a generic way and have
the PM core take care of them would be a godsend.

As for (b) driver-presence dependencies, I only know of a single
use case: Apparently the BCM57785 Ethernet/SDHC controller needs
the Ethernet driver to be loaded for the SDHC reader to work
(https://bugzilla.kernel.org/show_bug.cgi?id=73241#c56). I imagine
some power well is needed for SDHC and is only turned on when the
Ethernet driver is loaded. The two bit solution proposed above
would suffice for this use case: The two devices are siblings
and have no children themselves. (They're functions 0 and 1 of
a PCI endpoint device.)

Thanks,

Lukas

Re: [RFC/RFT][PATCH v2 0/7] Functional dependencies between devices

[Rafael J. Wysocki]

On Sat, Sep 10, 2016 at 1:39 PM, Lukas Wunner <lukas@wunner.de> wrote:
> On Thu, Sep 08, 2016 at 11:25:44PM +0200, Rafael J. Wysocki wrote:
>> As discussed in the recent "On-demand device probing" thread and in a
>> Kernel Summit session earlier today, there is a problem with handling
>> cases where functional dependencies between devices are involved.
>>
>> What I mean by a "functional dependency" is when the driver of device B
>> needs both device A and its driver to be present and functional to be
>> able to work.  This implies that the driver of A needs to be working
>> for B to be probed successfully and it cannot be unbound from the
>> device before the B's driver.  This also has certain consequences for
>> power management of these devices (suspend/resume and runtime PM
>> ordering).
>
> As a general observation, this series seems to conflate two
> separate issues:
>
> (a) non-hierarchical device dependencies
>     (a device depends on another device which is not its parent
>     with regards to (runtime) suspend/resume ordering)

You need to say what it means that one device depends on another one.
Without that you don't get anything useful.

> (b) driver-presence dependencies
>     (a device depends on another device to be bound to a driver
>     before it can probe / must be unbound before the other device
>     can unbind)
>
> Those two issues are orthogonal.

No, they aren't.

At least for a number of devices (quite likely it would be safe to say
that for the majority of them I think) the supplier can only provide
any useful service to its consumers when it has a driver bound to it
and running and the consumers cannot operate correctly without that
service.  That's why the "unbind consumers before unbinding the
supplier" logic is in there.

And in the context of this series a "dependency" boils down to the
ordering of execution of callbacks of certain type.  Like, for
example, "can I execute ->runtime_suspend() for device A before
executing it for device B?"  If there's a link between A and B where
the former is the supplier, the answer is "no".  The reason why is the
assumption that after ->runtime_suspend() A will stop responding and
will not be able to provide the service in question to B any more.

Likewise, if the driver of A goes away, this device will not be able
to provide the service in question to B any more, so it doesn't make
sense for the driver of B to still be running then.

This is the case I wanted to cover with this series if it was not clear before.

Of course, you don't like what device_links_unbind_consumers() does
because of your "Thunderbolt on Macs" use case, but guess what, you
can add a link flag to keep the consumer around when the supplier
driver goes away.  I just don't want that to be the default behavior.

> E.g. a driver-presence dependency may exist between a child and a
> parent, or between siblings, whereas a non-hierarchical device
> dependency by definition cannot exist between child/parent.
>
> Let's say I need a driver-presence dependency between parent/child.
> The PM core already guarantees correct (runtime) suspend/resume
> ordering between child. Device links duplicate that functionality.
> Oops?

No, they don't and that should be clear.

They are for out-of-the-tree dependencies only and the parent-child
one is *in* the tree.

Thanks,
Rafael

Re: [RFC/RFT][PATCH v2 0/7] Functional dependencies between devices

[Lukas Wunner]

On Sun, Sep 11, 2016 at 12:04:36AM +0200, Rafael J. Wysocki wrote:
> On Sat, Sep 10, 2016 at 1:39 PM, Lukas Wunner <lukas@wunner.de> wrote:
> > On Thu, Sep 08, 2016 at 11:25:44PM +0200, Rafael J. Wysocki wrote:
> >> As discussed in the recent "On-demand device probing" thread and in a
> >> Kernel Summit session earlier today, there is a problem with handling
> >> cases where functional dependencies between devices are involved.
> >>
> >> What I mean by a "functional dependency" is when the driver of device B
> >> needs both device A and its driver to be present and functional to be
> >> able to work.  This implies that the driver of A needs to be working
> >> for B to be probed successfully and it cannot be unbound from the
> >> device before the B's driver.  This also has certain consequences for
> >> power management of these devices (suspend/resume and runtime PM
> >> ordering).
> >
> > As a general observation, this series seems to conflate two
> > separate issues:
> >
> > (a) non-hierarchical device dependencies
> >     (a device depends on another device which is not its parent
> >     with regards to (runtime) suspend/resume ordering)
> 
> You need to say what it means that one device depends on another one.
> Without that you don't get anything useful.

As stated above, what it means is that "a device depends on another device
with regards to (runtime) suspend/resume ordering".


> > (b) driver-presence dependencies
> >     (a device depends on another device to be bound to a driver
> >     before it can probe / must be unbound before the other device
> >     can unbind)
> >
> > Those two issues are orthogonal.
> 
> No, they aren't.
> 
> At least for a number of devices (quite likely it would be safe to say
> that for the majority of them I think) the supplier can only provide

You got numbers to back up the majority claim?


> any useful service to its consumers when it has a driver bound to it
> and running and the consumers cannot operate correctly without that
> service.  That's why the "unbind consumers before unbinding the
> supplier" logic is in there.
> 
> And in the context of this series a "dependency" boils down to the
> ordering of execution of callbacks of certain type.  Like, for
> example, "can I execute ->runtime_suspend() for device A before
> executing it for device B?"  If there's a link between A and B where
> the former is the supplier, the answer is "no".  The reason why is the
> assumption that after ->runtime_suspend() A will stop responding and
> will not be able to provide the service in question to B any more.
> 
> Likewise, if the driver of A goes away, this device will not be able
> to provide the service in question to B any more, so it doesn't make
> sense for the driver of B to still be running then.
> 
> This is the case I wanted to cover with this series if it was not clear
> before.

That much was clear to me. I maintain that dependency with regards to
suspend/resume ordering and dependency with regards to driver presence
are two different things. The case you wanted to cover just happens to
be a combination of the two. The series would be more useful if either
type of dependency could be achieved without gratuitously also getting
the other type.

You could make the case that providing two separate APIs seems
wasteful since it would increase the amount of code and sometimes
both types of dependency are required at the same time anyway.
That would be a valid argument. But you should design the API thus
that either dependency type is optional.

Providing just dependency with regards to suspend/resume ordering
should be as simple as adding a "DEVICE_LINKS_DRIVER_DONT_CARE" flag.
Then the various calls that you're adding in dd.c would only have to
be called if the flag isn't set, same for the various places where
you check the link state.


> Of course, you don't like what device_links_unbind_consumers() does
> because of your "Thunderbolt on Macs" use case, but guess what,

To name a different use case: On hybrid graphics laptops, the discrete
GPU usually includes an HDA controller for external HDMI displays.
The GPU and the HDA controllers are siblings (functions 0 and 1 of a
PCI slot), yet in many laptops power is cut to both devices when _PS3
is executed for the GPU function. Currently we have a kludge where
the HDA controller is suspended before the GPU is powered down
(see vga_switcheroo_init_domain_pm_optimus_hdmi_audio()).

I envision the HDA controller to be a consumer of the GPU in those
cases, thus ensuring that it's suspended before power is cut.

The way your series works right now, the HDA controller can't be
bound unless the GPU is bound. That restriction is unnecessary.


I've also heard from the i915 folks that on some Intel chipsets, the
integrated HDA controller depends on power wells that are controlled
by the integrated GPU. I'm not sure if a driver for the GPU needs
to be present in this use case.


> you can add a link flag to keep the consumer around when the supplier
> driver goes away.  I just don't want that to be the default behavior.

I assume you're referring to the DEVICE_LINK_PERSISTENT flag and that
a separate patch is needed to make this work, as you've pointed out
in your e-mail of Sep 7.

According to the commit message, such links need to be added before
the consumer probes, which I fear will require addition of PCI quirks.
That's not very pretty, a DEVICE_LINKS_DRIVER_DONT_CARE flag would
seem preferrable.


> > E.g. a driver-presence dependency may exist between a child and a
> > parent, or between siblings, whereas a non-hierarchical device
> > dependency by definition cannot exist between child/parent.
> >
> > Let's say I need a driver-presence dependency between parent/child.
> > The PM core already guarantees correct (runtime) suspend/resume
> > ordering between child. Device links duplicate that functionality.
> > Oops?
> 
> No, they don't and that should be clear.
> 
> They are for out-of-the-tree dependencies only and the parent-child
> one is *in* the tree.

I'm sure there are situations where a driver presence dependency
is needed between parent/child and you should fully expect that
developers will try to employ device links for these use cases.
Which means that the code for suspend/resume device ordering is
executed twice.

Best regards,

Lukas

Re: [RFC/RFT][PATCH v2 0/7] Functional dependencies between devices

[Rafael J. Wysocki]

On Tuesday, September 13, 2016 07:57:31 PM Lukas Wunner wrote:
> On Sun, Sep 11, 2016 at 12:04:36AM +0200, Rafael J. Wysocki wrote:
> > On Sat, Sep 10, 2016 at 1:39 PM, Lukas Wunner <lukas@wunner.de> wrote:
> > > On Thu, Sep 08, 2016 at 11:25:44PM +0200, Rafael J. Wysocki wrote:
> > >> As discussed in the recent "On-demand device probing" thread and in a
> > >> Kernel Summit session earlier today, there is a problem with handling
> > >> cases where functional dependencies between devices are involved.
> > >>
> > >> What I mean by a "functional dependency" is when the driver of device B
> > >> needs both device A and its driver to be present and functional to be
> > >> able to work.  This implies that the driver of A needs to be working
> > >> for B to be probed successfully and it cannot be unbound from the
> > >> device before the B's driver.  This also has certain consequences for
> > >> power management of these devices (suspend/resume and runtime PM
> > >> ordering).
> > >
> > > As a general observation, this series seems to conflate two
> > > separate issues:
> > >
> > > (a) non-hierarchical device dependencies
> > >     (a device depends on another device which is not its parent
> > >     with regards to (runtime) suspend/resume ordering)
> > 
> > You need to say what it means that one device depends on another one.
> > Without that you don't get anything useful.
> 
> As stated above, what it means is that "a device depends on another device
> with regards to (runtime) suspend/resume ordering".

But this series isn't only about suspend/resume, which seems to be your point,
right?

> > > (b) driver-presence dependencies
> > >     (a device depends on another device to be bound to a driver
> > >     before it can probe / must be unbound before the other device
> > >     can unbind)
> > >
> > > Those two issues are orthogonal.
> > 
> > No, they aren't.
> > 
> > At least for a number of devices (quite likely it would be safe to say
> > that for the majority of them I think) the supplier can only provide
> 
> You got numbers to back up the majority claim?

This isn't a claim (which is why I said "quite likely" and so on).

I have examples, though.  I2C/SPI/GPIO controllers providing connections/pins
to devices that aren't their children, IOMMUs.

> > any useful service to its consumers when it has a driver bound to it
> > and running and the consumers cannot operate correctly without that
> > service.  That's why the "unbind consumers before unbinding the
> > supplier" logic is in there.
> > 
> > And in the context of this series a "dependency" boils down to the
> > ordering of execution of callbacks of certain type.  Like, for
> > example, "can I execute ->runtime_suspend() for device A before
> > executing it for device B?"  If there's a link between A and B where
> > the former is the supplier, the answer is "no".  The reason why is the
> > assumption that after ->runtime_suspend() A will stop responding and
> > will not be able to provide the service in question to B any more.
> > 
> > Likewise, if the driver of A goes away, this device will not be able
> > to provide the service in question to B any more, so it doesn't make
> > sense for the driver of B to still be running then.
> > 
> > This is the case I wanted to cover with this series if it was not clear
> > before.
> 
> That much was clear to me. I maintain that dependency with regards to
> suspend/resume ordering and dependency with regards to driver presence
> are two different things. The case you wanted to cover just happens to
> be a combination of the two. The series would be more useful if either
> type of dependency could be achieved without gratuitously also getting
> the other type.

It looks like you're referring to the previous iteration of the series here.

> You could make the case that providing two separate APIs seems
> wasteful since it would increase the amount of code and sometimes
> both types of dependency are required at the same time anyway.
> That would be a valid argument.

Well, that was my thinking, but I guess it wasn't clearly stated.

Maybe except that IMO it is "sometimes one type of the dependency is not
required" rather than the other way around.

> But you should design the API thus that either dependency type is optional.
> 
> Providing just dependency with regards to suspend/resume ordering
> should be as simple as adding a "DEVICE_LINKS_DRIVER_DONT_CARE" flag.
> Then the various calls that you're adding in dd.c would only have to
> be called if the flag isn't set, same for the various places where
> you check the link state.

OK, fair enough.  But that could be added later too.

> > Of course, you don't like what device_links_unbind_consumers() does
> > because of your "Thunderbolt on Macs" use case, but guess what,
> 
> To name a different use case: On hybrid graphics laptops, the discrete
> GPU usually includes an HDA controller for external HDMI displays.
> The GPU and the HDA controllers are siblings (functions 0 and 1 of a
> PCI slot), yet in many laptops power is cut to both devices when _PS3
> is executed for the GPU function. Currently we have a kludge where
> the HDA controller is suspended before the GPU is powered down
> (see vga_switcheroo_init_domain_pm_optimus_hdmi_audio()).
> 
> I envision the HDA controller to be a consumer of the GPU in those
> cases, thus ensuring that it's suspended before power is cut.

So this example isn't a good one IMO.  That clearly is a case when two
(or more) devices share power resources controlled by a single on/off
switch.  Which is a clear use case for a PM domain.

> The way your series works right now, the HDA controller can't be
> bound unless the GPU is bound. That restriction is unnecessary.

That depends on when the link is created, actually.  If the link is created
in the "active" state from the supplier probe, there's no such restriction
AFAICS.

The restriction is only there if (a) that is a permanent link and (b) it
was created before probing both the supplier and the consumer.

> I've also heard from the i915 folks that on some Intel chipsets, the
> integrated HDA controller depends on power wells that are controlled
> by the integrated GPU. I'm not sure if a driver for the GPU needs
> to be present in this use case.
> 
> 
> > you can add a link flag to keep the consumer around when the supplier
> > driver goes away.  I just don't want that to be the default behavior.
> 
> I assume you're referring to the DEVICE_LINK_PERSISTENT flag and that
> a separate patch is needed to make this work, as you've pointed out
> in your e-mail of Sep 7.

No, I'm not.  I was thinking about something like the DEVICE_LINKS_DRIVER_DONT_CARE
you mentioned above.  It would be easy enough to add on top of this series,
but I can put it in there if you insist.

> According to the commit message, such links need to be added before
> the consumer probes, which I fear will require addition of PCI quirks.
> That's not very pretty, a DEVICE_LINKS_DRIVER_DONT_CARE flag would
> seem preferrable.

The commit message may be outdated.

With the series as is you can create a link in (almost) any state from
anywhere.

> > > E.g. a driver-presence dependency may exist between a child and a
> > > parent, or between siblings, whereas a non-hierarchical device
> > > dependency by definition cannot exist between child/parent.
> > >
> > > Let's say I need a driver-presence dependency between parent/child.
> > > The PM core already guarantees correct (runtime) suspend/resume
> > > ordering between child. Device links duplicate that functionality.
> > > Oops?
> > 
> > No, they don't and that should be clear.
> > 
> > They are for out-of-the-tree dependencies only and the parent-child
> > one is *in* the tree.
> 
> I'm sure there are situations where a driver presence dependency
> is needed between parent/child and you should fully expect that
> developers will try to employ device links for these use cases.
> Which means that the code for suspend/resume device ordering is
> executed twice.

Creating a link between a parent and child would be a bug.  I guess
device_link_add() should just return NULL on an attempt to do that.

Thanks,
Rafael

Re: [RFC/RFT][PATCH v2 0/7] Functional dependencies between devices

[Lukas Wunner]

On Wed, Sep 14, 2016 at 01:18:16AM +0200, Rafael J. Wysocki wrote:
> On Tuesday, September 13, 2016 07:57:31 PM Lukas Wunner wrote:
> > To name a different use case: On hybrid graphics laptops, the discrete
> > GPU usually includes an HDA controller for external HDMI displays.
> > The GPU and the HDA controllers are siblings (functions 0 and 1 of a
> > PCI slot), yet in many laptops power is cut to both devices when _PS3
> > is executed for the GPU function. Currently we have a kludge where
> > the HDA controller is suspended before the GPU is powered down
> > (see vga_switcheroo_init_domain_pm_optimus_hdmi_audio()).
> > 
> > I envision the HDA controller to be a consumer of the GPU in those
> > cases, thus ensuring that it's suspended before power is cut.
> 
> So this example isn't a good one IMO.  That clearly is a case when two
> (or more) devices share power resources controlled by a single on/off
> switch.  Which is a clear use case for a PM domain.

TBH, I've never understood how a PM domain is supposed to solve this.
When power is cut at runtime for a struct dev_pm_domain, all devices
that were assigned this PM domain with dev_pm_domain_set() need to be
runtime suspended.  This requires that a list of devices is maintained
which were assigned the same PM domain, and that the PM domain's
->runtime_suspend hook isn't executed before all of these devices have
runtime_suspended.  Maybe I'm missing something but I don't see any code
to guarantee that in drivers/base/power/.  Rather, the PM domain's
->runtime_suspend hook is executed as soon as one of the devices in the
PM domain runtime suspends, *without* taking into consideration the
other devices in the PM domain.  They'll just be hanging in the air
with their device powered down.

>From what I've seen, people simply use struct dev_pm_domain as a way to
override the bus callbacks.  At least that's what Dave Airlie does in
vga_switcheroo.  But fundamentally that has nothing to do with shared
power resources, it only has to do with enforcing a different behaviour
than the bus.

Thus I don't understand what you mean if you say this is a use case for
a PM domain.


> > I'm sure there are situations where a driver presence dependency
> > is needed between parent/child and you should fully expect that
> > developers will try to employ device links for these use cases.
> > Which means that the code for suspend/resume device ordering is
> > executed twice.
> 
> Creating a link between a parent and child would be a bug.  I guess
> device_link_add() should just return NULL on an attempt to do that.

To be clear, while linking a parent (as consumer) to a child
(as supplier) needs to be prevented since it introduces a dependency
loop, the converse should IMO be allowed.

That would be the case when someone needs a driver presence dependency,
but doesn't need a suspend/resume ordering dependency (because it's
already guaranteed by the PM core for parent/child).  In that case the
child will simultaneously be a consumer, which means e.g. that dpm_wait()
will be executed twice for the same device, but that overhead is probably
negligible.

Thanks,

Lukas

Re: [RFC/RFT][PATCH v2 0/7] Functional dependencies between devices

[Marek Szyprowski]

Hi Rafael,


On 2016-09-08 23:25, Rafael J. Wysocki wrote:
> Hi Everyone,
>
> This is a refresh of the functional dependencies series that I posted last
> year and which has picked up by Marek quite recently.  For reference, appended
> is my introductory message sent previously (which may be slightly outdated now).
>
> As last time, the first patch rearranges the code around __device_release_driver()
> a bit to prepare it for the next one (it actually hasn't changed AFAICS).
>
> The second patch introduces the actual device links mechanics, but without
> system suspend/resume and runtime PM support which are added by the subsequent
> patches.
>
> Some bugs found by Marek during his work on these patches should be fixed
> here.  In particular, the endless recursion in device_reorder_to_tail()
> which simply was broken before.
>
> There are two additional patches to address the issue with runtime PM support
> that occured when runtime PM was disabled for some suppliers due to a PM
> sleep transition in progress.  Those patches simply make runtime PM helpers
> return 0 in that case which may be controversial, so please let me know if
> there are concerns about those.
>
> The way device_link_add() works is a bit different, as it takes an additional
> status argument now.  That makes it possible to create a link in any state,
> with extra care of course, and should address the problem pointed to by Lukas
> during the previous discussion.
>
> Also some comments from Tomeu have been addressed.
>
> This hasn't been really tested yet and I'm sort of relying on Marek to test
> it, because he has a use case ready.  Hence, the RFT tag on the series.

I've checked it with my updated Exynos SYSMMU patches and it works 
really well.
Both runtime pm and system sleep. It took some time to do the test, because
this patchset changed somehow the probe order of the devices, what revealed
some issues related to incorrect driver and device registration in Samsung
Exynos FIMC-IS driver, but those were not related to Rafael's changes.
I will post my patches in a few minutes.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>

Rafael, BTW, didn't you plan to change the name of the device_link_add()
function to device_dependency_add() to avoid confusion with network device
"link"?

> Overall, please let me know what you think.
>
> Thanks,
> Rafael
>
>
> Introduction:
>
> As discussed in the recent "On-demand device probing" thread and in a Kernel
> Summit session earlier today, there is a problem with handling cases where
> functional dependencies between devices are involved.
>
> What I mean by a "functional dependency" is when the driver of device B needs
> both device A and its driver to be present and functional to be able to work.
> This implies that the driver of A needs to be working for B to be probed
> successfully and it cannot be unbound from the device before the B's driver.
> This also has certain consequences for power management of these devices
> (suspend/resume and runtime PM ordering).
>
> So I want to be able to represent those functional dependencies between devices
> and I'd like the driver core to track them and act on them in certain cases
> where they matter.  The argument for doing that in the driver core is that
> there are quite a few distinct use cases related to that, they are relatively
> hard to get right in a driver (if one wants to address all of them properly)
> and it only gets worse if multiplied by the number of drivers potentially
> needing to do it.  Morever, at least one case (asynchronous system suspend/resume)
> cannot be handled in a single driver at all, because it requires the driver of A
> to wait for B to suspend (during system suspend) and the driver of B to wait for
> A to resume (during system resume).
>
> My idea is to represent a supplier-consumer dependency between devices (or
> more precisely between device+driver combos) as a "link" object containing
> pointers to the devices in question, a list node for each of them and some
> additional information related to the management of those objects, ie.
> something like:
>
> struct device_link {
>          struct device *supplier;
>          struct list_head supplier_node;
>          struct device *consumer;
>          struct list_head consumer_node;
>          <flags, status etc>
> };
>
> In general, there will be two lists of those things per device, one list
> of links to consumers and one list of links to suppliers.
>
> In that picture, links will be created by calling, say:
>
> int device_add_link(struct device *me, struct device *my_supplier, unsigned int flags);
>
> and they will be deleted by the driver core when not needed any more.  The
> creation of a link should also cause dpm_list and the list used during shutdown
> to be reordered if needed.
>
> In principle, it seems usefult to consider two types of links, one created
> at device registration time (when registering the second device from the linked
> pair, whichever it is) and one created at probe time (of the consumer device).
> I'll refer to them as "permanent" and "probe-time" links, respectively.
>
> The permanent links (created at device registration time) will stay around
> until one of the linked devices is unregistered (at which time the driver
> core will drop the link along with the device going away).  The probe-time
> ones will be dropped (automatically) at the consumer device driver unbind time.
>
> There's a question about what if the supplier device is being unbound before
> the consumer one (for example, as a result of a hotplug event).  My current
> view on that is that the consumer needs to be force-unbound in that case too,
> but I guess I may be persuaded otherwise given sufficiently convincing
> arguments.  Anyway, there are reasons to do that, like for example it may
> help with the synchronization.  Namely, if there's a rule that suppliers
> cannot be unbound before any consumers linked to them, than the list of links
> to suppliers for a consumer can only change at its registration/probe or
> unbind/remove times (which simplifies things quite a bit).
>
> With that, the permanent links existing at the probe time for a consumer
> device can be used to check whether or not to defer the probing of it
> even before executing its probe callback.  In turn, system suspend
> synchronization should be a matter of calling device_pm_wait_for_dev()
> for all consumers of a supplier device, in analogy with dpm_wait_for_children(),
> and so on.
>
> Of course, the new lists have to be stable during those operations and ensuring
> that is going to be somewhat tricky (AFAICS right now at least), but apart from
> that the whole concept looks reasonably straightforward to me.
>
>
>

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [RFC/RFT][PATCH v2 0/7] Functional dependencies between devices

[Rafael J. Wysocki]

On Tuesday, September 13, 2016 11:58:56 AM Marek Szyprowski wrote:
> Hi Rafael,
> 
> 
> On 2016-09-08 23:25, Rafael J. Wysocki wrote:
> > Hi Everyone,
> >
> > This is a refresh of the functional dependencies series that I posted last
> > year and which has picked up by Marek quite recently.  For reference, appended
> > is my introductory message sent previously (which may be slightly outdated now).
> >
> > As last time, the first patch rearranges the code around __device_release_driver()
> > a bit to prepare it for the next one (it actually hasn't changed AFAICS).
> >
> > The second patch introduces the actual device links mechanics, but without
> > system suspend/resume and runtime PM support which are added by the subsequent
> > patches.
> >
> > Some bugs found by Marek during his work on these patches should be fixed
> > here.  In particular, the endless recursion in device_reorder_to_tail()
> > which simply was broken before.
> >
> > There are two additional patches to address the issue with runtime PM support
> > that occured when runtime PM was disabled for some suppliers due to a PM
> > sleep transition in progress.  Those patches simply make runtime PM helpers
> > return 0 in that case which may be controversial, so please let me know if
> > there are concerns about those.
> >
> > The way device_link_add() works is a bit different, as it takes an additional
> > status argument now.  That makes it possible to create a link in any state,
> > with extra care of course, and should address the problem pointed to by Lukas
> > during the previous discussion.
> >
> > Also some comments from Tomeu have been addressed.
> >
> > This hasn't been really tested yet and I'm sort of relying on Marek to test
> > it, because he has a use case ready.  Hence, the RFT tag on the series.
> 
> I've checked it with my updated Exynos SYSMMU patches and it works 
> really well.
> Both runtime pm and system sleep. It took some time to do the test, because
> this patchset changed somehow the probe order of the devices, what revealed
> some issues related to incorrect driver and device registration in Samsung
> Exynos FIMC-IS driver, but those were not related to Rafael's changes.
> I will post my patches in a few minutes.
> 
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>

Thanks, much appreciated!

> Rafael, BTW, didn't you plan to change the name of the device_link_add()
> function to device_dependency_add() to avoid confusion with network device
> "link"?

I was concerned about the "devlink" name in particular, but I thought that
struct device_link would be distinct enough.  If not, I can still change it.

Thanks,
Rafael

Re: [RFC/RFT][PATCH v2 0/7] Functional dependencies between devices

[Lukas Wunner]

On Wed, Sep 14, 2016 at 12:41:33AM +0200, Rafael J. Wysocki wrote:
> On Tuesday, September 13, 2016 11:58:56 AM Marek Szyprowski wrote:
> > Rafael, BTW, didn't you plan to change the name of the device_link_add()
> > function to device_dependency_add() to avoid confusion with network device
> > "link"?
> 
> I was concerned about the "devlink" name in particular, but I thought that
> struct device_link would be distinct enough.  If not, I can still change it.

The "links" term has already been established, in a sense, by commit
5063ce1571b7 ("PM / Domains: Allow generic PM domains to have multiple
masters").

However the genpd links introduced by that commit use a master/slave
terminology, rather than supplier/consumer.

Best regards,

Lukas

[RFC/RFT][PATCH v3 0/5] Functional dependencies between devices

[Rafael J. Wysocki]

Hi Everyone,

On Thursday, September 08, 2016 11:25:44 PM Rafael J. Wysocki wrote:
> Hi Everyone,
> 
> This is a refresh of the functional dependencies series that I posted last
> year and which has picked up by Marek quite recently.  For reference, appended
> is my introductory message sent previously (which may be slightly outdated now).
> 
> As last time, the first patch rearranges the code around __device_release_driver()
> a bit to prepare it for the next one (it actually hasn't changed AFAICS).
> 
> The second patch introduces the actual device links mechanics, but without
> system suspend/resume and runtime PM support which are added by the subsequent
> patches.
> 
> Some bugs found by Marek during his work on these patches should be fixed
> here.  In particular, the endless recursion in device_reorder_to_tail()
> which simply was broken before.
> 
> There are two additional patches to address the issue with runtime PM support
> that occured when runtime PM was disabled for some suppliers due to a PM
> sleep transition in progress.  Those patches simply make runtime PM helpers
> return 0 in that case which may be controversial, so please let me know if
> there are concerns about those.
> 
> The way device_link_add() works is a bit different, as it takes an additional
> status argument now.  That makes it possible to create a link in any state,
> with extra care of course, and should address the problem pointed to by Lukas
> during the previous discussion.
> 
> Also some comments from Tomeu have been addressed.

An update here.

The first patch hasn't changed, so I'm resending it.

The majority of changes in the other patches are in order to address Lukas'
comments.

First off, I added a DEVICE_LINK_STATELESS flag that will prevent the driver
core from trying to maintain device links having it set.

Also, the DEVICE_LINK_PERSISTENT flag was dropped (as link "persistence" is the
default behavior now) and there's a new one, DEVICE_LINK_AUTOREMOVE, that will
cause the driver core to remove the link on the consumer driver unbind.

Moreover, the code checks attempts to create a link between a parent and a child
device now and actively prevents that from happening.

The changelog of the second patch has been updated as requested by Ulf.

The third patch was updated to fix a bug related to the (previously missing)
clearing of power.direct_complete for supplier devices having consumers that
don't use direct_complete.

The next two (runtime PM) patches turned out to be unnecessary, so I've dropped
them.

The runtime PM patch [4/5] was reorganized somewhat to reduce the indentation
level in there, but the code flow introduced by it is essentially the same
and the last patch was simply rebased on top of the new series.

If this version still works for Marek, I'll probably drop the RFC tag from it
in the next iteration.

Thanks,
Rafael

[Resend][RFC/RFT][PATCH v3 1/5] driver core: Add a wrapper around __device_release_driver()

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Add an internal wrapper around __device_release_driver() that will
acquire device locks and do the necessary checks before calling it.

The next patch will make use of it.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/base/dd.c | 30 ++++++++++++++++++------------
 1 file changed, 18 insertions(+), 12 deletions(-)

diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index 16688f50729c..d9e76e9205c7 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -796,6 +796,22 @@ static void __device_release_driver(struct device *dev)
 	}
 }
 
+static void device_release_driver_internal(struct device *dev,
+					   struct device_driver *drv,
+					   struct device *parent)
+{
+	if (parent)
+		device_lock(parent);
+
+	device_lock(dev);
+	if (!drv || drv == dev->driver)
+		__device_release_driver(dev);
+
+	device_unlock(dev);
+	if (parent)
+		device_unlock(parent);
+}
+
 /**
  * device_release_driver - manually detach device from driver.
  * @dev: device.
@@ -810,9 +826,7 @@ void device_release_driver(struct device *dev)
 	 * within their ->remove callback for the same device, they
 	 * will deadlock right here.
 	 */
-	device_lock(dev);
-	__device_release_driver(dev);
-	device_unlock(dev);
+	device_release_driver_internal(dev, NULL, NULL);
 }
 EXPORT_SYMBOL_GPL(device_release_driver);
 
@@ -837,15 +851,7 @@ void driver_detach(struct device_driver *drv)
 		dev = dev_prv->device;
 		get_device(dev);
 		spin_unlock(&drv->p->klist_devices.k_lock);
-
-		if (dev->parent)	/* Needed for USB */
-			device_lock(dev->parent);
-		device_lock(dev);
-		if (dev->driver == drv)
-			__device_release_driver(dev);
-		device_unlock(dev);
-		if (dev->parent)
-			device_unlock(dev->parent);
+		device_release_driver_internal(dev, drv, dev->parent);
 		put_device(dev);
 	}
 }

[RFC/RFT][PATCH v3 2/5] driver core: Functional dependencies tracking support

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Currently, there is a problem with taking functional dependencies
between into account.

What I mean by a "functional dependency" is when the driver of device
B needs device A to be functional and (generally) its driver to be
present in order to work properly.  This has certain consequences
for power management (suspend/resume and runtime PM ordering) and
shutdown ordering of these devices.  In general, it also implies that
the driver of A needs to be working for B to be probed successfully
and it cannot be unbound from the device before the B's driver.  

Support for representing those functional dependencies between
devices is added here to allow the driver core to track them and act
on them in certain cases where applicable.

The argument for doing that in the driver core is that there are
quite a few distinct use cases involving device dependencies, they
are relatively hard to get right in a driver (if one wants to
address all of them properly) and it only gets worse if multiplied
by the number of drivers potentially needing to do it.  Morever, at
least one case (asynchronous system suspend/resume) cannot be handled
in a single driver at all, because it requires the driver of A to
wait for B to suspend (during system suspend) and the driver of B to
wait for A to resume (during system resume).

For this reason, represent dependencies between devices as "links",
with the help of struct device_link objects each containing pointers
to the "linked" devices, a list node for each of them, status
information, flags, a lock and an RCU head for synchronization.

Also add two new list heads, links_to_consumers and links_to_suppliers,
to struct device to represent the lists of links to the devices that
depend on the given one (consumers) and to the devices depended on
by it (suppliers), respectively.

The entire data structure consisting of all of the lists of link
objects for all devices is protected by SRCU (for list walking)
and a by mutex (for link object addition/removal).  In addition
to that, each link object has an internal status field whose
value reflects what's happening to the devices pointed to by
the link.  That status field is protected by an internal spinlock.

New links are added by calling device_link_add() which takes four
arguments: pointers to the devices in question, the initial status
of the link and flags.  In particular, if DEVICE_LINK_STATELESS is
set in the flags, the link status is not to be taken into account
for this link and the driver core will not manage it.  In turn, if
DEVICE_LINK_AUTOREMOVE is set in the flags, the driver core will
remove the link automatically when the consumer device driver
unbinds from it.

One of the actions carried out by device_link_add() is to reorder
the lists used for device shutdown and system suspend/resume to
put the consumer device along with all of its children and all of
its consumers (and so on, recursively) to the ends of those list
in order to ensure the right ordering between all of the supplier
and consumer devices.

For this reason, it is not possible to create a link between two
devices if the would-be supplier device already depends on the
would-be consumer device as either a direct descendant of it or a
consumer of one of its direct descendants or one of its consumers
and so on.

It also is impossible to create a link between a parent and a child
device (in any direction).

There are two types of link objects, persistent and non-persistent.
The persistent ones stay around until one of the target devices is
deleted, while the non-persistent ones are removed automatically when
the consumer driver unbinds from its device (ie. they are assumed to
be valid only as long as the consumer device has a driver bound to
it).  Persistent links are created by default and non-persistent
links are created when the DEVICE_LINK_AUTOREMOVE flag is passed
to device_link_add().

Both persistent and non-persistent device links can be deleted
explicitly with the help of device_link_del().

Links created without the DEVICE_LINK_STATELESS flag set are managed
by the driver core using a simple state machine.  There are 5 states
each link can be in: DORMANT (unused), AVAILABLE (the supplier driver
is present and functional), CONSUMER_PROBE (the consumer driver is
probing), ACTIVE (both supplier and consumer drivers are present and
functional), and SUPPLIER_UNBIND (the supplier driver is unbinding).
The driver core updates the link state automatically depending on
what happens to the linked devices and for each link state specific
actions are taken in addition to that.

For example, if the supplier driver unbinds from its device, the
driver core will also unbind the drivers of all of its consumers
automatically under the assumption that they cannot function
properly without the supplier.  Analogously, the driver core will
only allow the consumer driver to bind to its device is the
supplier driver is present and functional (ie. the link is in
the AVAILABLE state).  If that's not the case, it will rely on
the existing deferred probing mechanism to wait for the supplier
driver to become available.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/base/base.h    |   11 +
 drivers/base/core.c    |  471 +++++++++++++++++++++++++++++++++++++++++++++++++
 drivers/base/dd.c      |   42 +++-
 include/linux/device.h |   38 +++
 4 files changed, 557 insertions(+), 5 deletions(-)

Index: linux-pm/drivers/base/base.h
===================================================================
--- linux-pm.orig/drivers/base/base.h
+++ linux-pm/drivers/base/base.h
@@ -107,6 +107,9 @@ extern void bus_remove_device(struct dev
 
 extern int bus_add_driver(struct device_driver *drv);
 extern void bus_remove_driver(struct device_driver *drv);
+extern void device_release_driver_internal(struct device *dev,
+					   struct device_driver *drv,
+					   struct device *parent);
 
 extern void driver_detach(struct device_driver *drv);
 extern int driver_probe_device(struct device_driver *drv, struct device *dev);
@@ -152,3 +155,11 @@ extern int devtmpfs_init(void);
 #else
 static inline int devtmpfs_init(void) { return 0; }
 #endif
+
+/* Device links */
+extern int device_links_check_suppliers(struct device *dev);
+extern void device_links_driver_bound(struct device *dev);
+extern void device_links_driver_gone(struct device *dev);
+extern void device_links_no_driver(struct device *dev);
+extern bool device_links_busy(struct device *dev);
+extern void device_links_unbind_consumers(struct device *dev);
Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -44,6 +44,450 @@ static int __init sysfs_deprecated_setup
 early_param("sysfs.deprecated", sysfs_deprecated_setup);
 #endif
 
+/* Device links support. */
+
+DEFINE_STATIC_SRCU(device_links_srcu);
+static DEFINE_MUTEX(device_links_lock);
+
+/**
+ * device_is_dependent - Check if one device depends on another one
+ * @dev: Device to check dependencies for.
+ * @target: Device to check against.
+ *
+ * Check if @target depends on @dev or any device dependent on it (its child or
+ * its consumer etc).  Return 1 if that is the case or 0 otherwise.
+ */
+static int device_is_dependent(struct device *dev, void *target)
+{
+	struct device_link *link;
+	int ret;
+
+	if (WARN_ON(dev == target))
+		return 1;
+
+	ret = device_for_each_child(dev, target, device_is_dependent);
+	if (ret)
+		return ret;
+
+	list_for_each_entry(link, &dev->links_to_consumers, s_node) {
+		if (WARN_ON(link->consumer == target))
+			return 1;
+
+		ret = device_is_dependent(link->consumer, target);
+		if (ret)
+			break;
+	}
+	return ret;
+}
+
+static int device_reorder_to_tail(struct device *dev, void *not_used)
+{
+	struct device_link *link;
+
+	devices_kset_move_last(dev);
+	device_pm_move_last(dev);
+	device_for_each_child(dev, NULL, device_reorder_to_tail);
+	list_for_each_entry(link, &dev->links_to_consumers, s_node)
+		device_reorder_to_tail(link->consumer, NULL);
+
+	return 0;
+}
+
+/**
+ * device_link_add - Create a link between two devices.
+ * @consumer: Consumer end of the link.
+ * @supplier: Supplier end of the link.
+ * @status: The initial status of the link.
+ * @flags: Link flags.
+ *
+ * If the DEVICE_LINK_STATELESS flag is set, @status is ignored.  Otherwise,
+ * the caller is responsible for ensuring that @status reflects the current
+ * status of both @consumer and @supplier.
+ *
+ * If the DEVICE_LINK_AUTOREMOVE is set, the link will be removed automatically
+ * when the consumer device driver unbinds from it.  The combination of both
+ * DEVICE_LINK_AUTOREMOVE and DEVICE_LINK_STATELESS set is invalid and will
+ * cause NULL to be returned.
+ *
+ * A side effect of the link creation is re-ordering of dpm_list and the
+ * devices_kset list by moving the consumer device and all devices depending
+ * on it to the ends of these lists.
+ */
+struct device_link *device_link_add(struct device *consumer,
+				    struct device *supplier,
+				    enum device_link_status status, u32 flags)
+{
+	struct device_link *link;
+
+	if (!consumer || !supplier || supplier == consumer->parent ||
+	    ((flags & DEVICE_LINK_STATELESS) && (flags & DEVICE_LINK_AUTOREMOVE)))
+		return NULL;
+
+	mutex_lock(&device_links_lock);
+
+	/*
+	 * If there is a reverse dependency between the consumer and the
+	 * supplier already in the graph, return NULL.
+	 */
+	if (device_is_dependent(consumer, supplier)) {
+		link = NULL;
+		goto out;
+	}
+
+	list_for_each_entry(link, &supplier->links_to_consumers, s_node)
+		if (link->consumer == consumer)
+			goto out;
+
+	link = kmalloc(sizeof(*link), GFP_KERNEL);
+	if (!link)
+		goto out;
+
+	get_device(supplier);
+	link->supplier = supplier;
+	INIT_LIST_HEAD(&link->s_node);
+	get_device(consumer);
+	link->consumer = consumer;
+	INIT_LIST_HEAD(&link->c_node);
+	spin_lock_init(&link->lock);
+	link->flags = flags;
+	link->status = (flags & DEVICE_LINK_STATELESS) ?
+					DEVICE_LINK_NO_STATE : status;
+
+	/*
+	 * Move the consumer and all of the devices depending on it to the end
+	 * of dpm_list and the devices_kset list.
+	 *
+	 * It is necessary to hold dpm_list locked throughout all that or else
+	 * we may end up suspending with a wrong ordering of it.
+	 */
+	device_pm_lock();
+	device_reorder_to_tail(consumer, NULL);
+	device_pm_unlock();
+
+	list_add_tail_rcu(&link->s_node, &supplier->links_to_consumers);
+	list_add_tail_rcu(&link->c_node, &consumer->links_to_suppliers);
+
+	dev_info(consumer, "Linked as a consumer to %s\n", dev_name(supplier));
+
+ out:
+	mutex_unlock(&device_links_lock);
+	return link;
+}
+EXPORT_SYMBOL_GPL(device_link_add);
+
+static void __device_link_free_srcu(struct rcu_head *rhead)
+{
+	struct device_link *link;
+
+	link = container_of(rhead, struct device_link, rcu_head);
+	put_device(link->consumer);
+	put_device(link->supplier);
+	kfree(link);
+}
+
+static void __device_link_del(struct device_link *link)
+{
+	dev_info(link->consumer, "Dropping the link to %s\n",
+		 dev_name(link->supplier));
+
+	list_del_rcu(&link->s_node);
+	list_del_rcu(&link->c_node);
+	call_srcu(&device_links_srcu, &link->rcu_head, __device_link_free_srcu);
+}
+
+/**
+ * device_link_del - Delete a link between two devices.
+ * @link: Device link to delete.
+ *
+ * The caller must ensure proper synchronization of this function with runtime
+ * PM.
+ */
+void device_link_del(struct device_link *link)
+{
+	mutex_lock(&device_links_lock);
+	device_pm_lock();
+	__device_link_del(link);
+	device_pm_unlock();
+	mutex_unlock(&device_links_lock);
+}
+EXPORT_SYMBOL_GPL(device_link_del);
+
+static int device_links_read_lock(void)
+{
+	return srcu_read_lock(&device_links_srcu);
+}
+
+static void device_links_read_unlock(int idx)
+{
+	return srcu_read_unlock(&device_links_srcu, idx);
+}
+
+static void device_links_missing_supplier(struct device *dev)
+{
+	struct device_link *link;
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
+		spin_lock(&link->lock);
+
+		if (link->status == DEVICE_LINK_CONSUMER_PROBE)
+			link->status = DEVICE_LINK_AVAILABLE;
+
+		spin_unlock(&link->lock);
+	}
+}
+
+/**
+ * device_links_check_suppliers - Check supplier devices for this one.
+ * @dev: Consumer device.
+ *
+ * Check links from this device to any suppliers.  Walk the list of the device's
+ * consumer links and see if all of the suppliers are available.  If not, simply
+ * return -EPROBE_DEFER.
+ *
+ * Walk the list under SRCU and check each link's status field under its lock.
+ *
+ * We need to guarantee that the supplier will not go away after the check has
+ * been positive here.  It only can go away in __device_release_driver() and
+ * that function  checks the device's links to consumers.  This means we need to
+ * mark the link as "consumer probe in progress" to make the supplier removal
+ * wait for us to complete (or bad things may happen).
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ */
+int device_links_check_suppliers(struct device *dev)
+{
+	struct device_link *link;
+	int idx, ret = 0;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		spin_lock(&link->lock);
+		if (link->status != DEVICE_LINK_AVAILABLE) {
+			spin_unlock(&link->lock);
+			device_links_missing_supplier(dev);
+			ret = -EPROBE_DEFER;
+			break;
+		}
+		link->status = DEVICE_LINK_CONSUMER_PROBE;
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+	return ret;
+}
+
+/**
+ * device_links_driver_bound - Update device links after probing its driver.
+ * @dev: Device to update the links for.
+ *
+ * The probe has been successful, so update links from this device to any
+ * consumers by changing their status to "available".
+ *
+ * Also change the status of @dev's links to suppliers to "active".
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ */
+void device_links_driver_bound(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		spin_lock(&link->lock);
+		WARN_ON(link->status != DEVICE_LINK_DORMANT);
+		link->status = DEVICE_LINK_AVAILABLE;
+		spin_unlock(&link->lock);
+	}
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		spin_lock(&link->lock);
+		WARN_ON(link->status != DEVICE_LINK_CONSUMER_PROBE);
+		link->status = DEVICE_LINK_ACTIVE;
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+}
+
+/**
+ * device_links_driver_gone - Update links after driver removal.
+ * @dev: Device whose driver has gone away.
+ *
+ * Update links to consumers for @dev by changing their status to "dormant".
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ */
+void device_links_driver_gone(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		WARN_ON(link->flags & DEVICE_LINK_AUTOREMOVE);
+		spin_lock(&link->lock);
+		WARN_ON(link->status != DEVICE_LINK_SUPPLIER_UNBIND);
+		link->status = DEVICE_LINK_DORMANT;
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+}
+
+/**
+ * device_links_no_driver - Update links of a device without a driver.
+ * @dev: Device without a drvier.
+ *
+ * Delete all non-persistent links from this device to any suppliers.
+ *
+ * Persistent links stay around, but their status is changed to "available",
+ * unless they already are in the "supplier unbind in progress" state in which
+ * case they need not be updated.
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ */
+void device_links_no_driver(struct device *dev)
+{
+	struct device_link *link, *ln;
+
+	mutex_lock(&device_links_lock);
+
+	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_suppliers, c_node)
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		if (link->flags & DEVICE_LINK_AUTOREMOVE) {
+			__device_link_del(link);
+		} else {
+			spin_lock(&link->lock);
+
+			if (link->status != DEVICE_LINK_SUPPLIER_UNBIND)
+				link->status = DEVICE_LINK_AVAILABLE;
+
+			spin_unlock(&link->lock);
+		}
+
+	mutex_unlock(&device_links_lock);
+}
+
+/**
+ * device_links_busy - Check if there are any busy links to consumers.
+ * @dev: Device to check.
+ *
+ * Check each consumer of the device and return 'true' if its link's status
+ * is one of "consumer probe" or "active" (meaning that the given consumer is
+ * probing right now or its driver is present).  Otherwise, change the link
+ * state to "supplier unbind" to prevent the consumer from being probed
+ * successfully going forward.
+ *
+ * Return 'false' if there are no probing or active consumers.
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ */
+bool device_links_busy(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+	bool ret = false;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		spin_lock(&link->lock);
+		if (link->status == DEVICE_LINK_CONSUMER_PROBE
+		    || link->status == DEVICE_LINK_ACTIVE) {
+			spin_unlock(&link->lock);
+			ret = true;
+			break;
+		}
+		link->status = DEVICE_LINK_SUPPLIER_UNBIND;
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+	return ret;
+}
+
+/**
+ * device_links_unbind_consumers - Force unbind consumers of the given device.
+ * @dev: Device to unbind the consumers of.
+ *
+ * Walk the list of links to consumers for @dev and if any of them is in the
+ * "consumer probe" state, wait for all device probes in progress to complete
+ * and start over.
+ *
+ * If that's not the case, change the status of the link to "supplier unbind"
+ * and check if the link was in the "active" state.  If so, force the consumer
+ * driver to unbind and start over (the consumer will not re-probe as we have
+ * changed the state of the link already).
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ */
+void device_links_unbind_consumers(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+ start:
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
+		enum device_link_status status;
+
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		spin_lock(&link->lock);
+		status = link->status;
+		if (status == DEVICE_LINK_CONSUMER_PROBE) {
+			spin_unlock(&link->lock);
+
+			device_links_read_unlock(idx);
+
+			wait_for_device_probe();
+			goto start;
+		}
+		link->status = DEVICE_LINK_SUPPLIER_UNBIND;
+		if (status == DEVICE_LINK_ACTIVE) {
+			struct device *consumer = link->consumer;
+
+			get_device(consumer);
+			spin_unlock(&link->lock);
+
+			device_links_read_unlock(idx);
+
+			device_release_driver_internal(consumer, NULL,
+						       consumer->parent);
+			put_device(consumer);
+			goto start;
+		}
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+}
+
+/* Device links support end. */
+
 int (*platform_notify)(struct device *dev) = NULL;
 int (*platform_notify_remove)(struct device *dev) = NULL;
 static struct kobject *dev_kobj;
@@ -711,6 +1155,8 @@ void device_initialize(struct device *de
 #ifdef CONFIG_GENERIC_MSI_IRQ
 	INIT_LIST_HEAD(&dev->msi_list);
 #endif
+	INIT_LIST_HEAD(&dev->links_to_consumers);
+	INIT_LIST_HEAD(&dev->links_to_suppliers);
 }
 EXPORT_SYMBOL_GPL(device_initialize);
 
@@ -1233,6 +1679,7 @@ void device_del(struct device *dev)
 {
 	struct device *parent = dev->parent;
 	struct class_interface *class_intf;
+	struct device_link *link, *ln;
 
 	/* Notify clients of device removal.  This call must come
 	 * before dpm_sysfs_remove().
@@ -1240,6 +1687,30 @@ void device_del(struct device *dev)
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DEL_DEVICE, dev);
+
+	/*
+	 * Delete all of the remaining links from this device to any other
+	 * devices (either consumers or suppliers).
+	 *
+	 * This requires that all links be dormant, so warn if that's no the
+	 * case.
+	 */
+	mutex_lock(&device_links_lock);
+
+	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_suppliers, c_node) {
+		WARN_ON(link->status != DEVICE_LINK_DORMANT &&
+			!(link->flags & DEVICE_LINK_STATELESS));
+		__device_link_del(link);
+	}
+
+	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_consumers, s_node) {
+		WARN_ON(link->status != DEVICE_LINK_DORMANT &&
+			!(link->flags & DEVICE_LINK_STATELESS));
+		__device_link_del(link);
+	}
+
+	mutex_unlock(&device_links_lock);
+
 	dpm_sysfs_remove(dev);
 	if (parent)
 		klist_del(&dev->p->knode_parent);
Index: linux-pm/drivers/base/dd.c
===================================================================
--- linux-pm.orig/drivers/base/dd.c
+++ linux-pm/drivers/base/dd.c
@@ -249,6 +249,7 @@ static void driver_bound(struct device *
 		 __func__, dev_name(dev));
 
 	klist_add_tail(&dev->p->knode_driver, &dev->driver->p->klist_devices);
+	device_links_driver_bound(dev);
 
 	device_pm_check_callbacks(dev);
 
@@ -399,6 +400,7 @@ probe_failed:
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
 pinctrl_bind_failed:
+	device_links_no_driver(dev);
 	devres_release_all(dev);
 	driver_sysfs_remove(dev);
 	dev->driver = NULL;
@@ -489,6 +491,10 @@ int driver_probe_device(struct device_dr
 	if (!device_is_registered(dev))
 		return -ENODEV;
 
+	ret = device_links_check_suppliers(dev);
+	if (ret)
+		return ret;
+
 	pr_debug("bus: '%s': %s: matched device %s with driver %s\n",
 		 drv->bus->name, __func__, dev_name(dev), drv->name);
 
@@ -756,7 +762,7 @@ EXPORT_SYMBOL_GPL(driver_attach);
  * __device_release_driver() must be called with @dev lock held.
  * When called for a USB interface, @dev->parent lock must be held as well.
  */
-static void __device_release_driver(struct device *dev)
+static void __device_release_driver(struct device *dev, struct device *parent)
 {
 	struct device_driver *drv;
 
@@ -765,6 +771,25 @@ static void __device_release_driver(stru
 		if (driver_allows_async_probing(drv))
 			async_synchronize_full();
 
+		while (device_links_busy(dev)) {
+			device_unlock(dev);
+			if (parent)
+				device_unlock(parent);
+
+			device_links_unbind_consumers(dev);
+			if (parent)
+				device_lock(parent);
+
+			device_lock(dev);
+			/*
+			 * A concurrent invocation of the same function might
+			 * have released the driver successfully while this one
+			 * was waiting, so check for that.
+			 */
+			if (dev->driver != drv)
+				return;
+		}
+
 		pm_runtime_get_sync(dev);
 
 		driver_sysfs_remove(dev);
@@ -780,6 +805,9 @@ static void __device_release_driver(stru
 			dev->bus->remove(dev);
 		else if (drv->remove)
 			drv->remove(dev);
+
+		device_links_driver_gone(dev);
+		device_links_no_driver(dev);
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);
@@ -796,16 +824,16 @@ static void __device_release_driver(stru
 	}
 }
 
-static void device_release_driver_internal(struct device *dev,
-					   struct device_driver *drv,
-					   struct device *parent)
+void device_release_driver_internal(struct device *dev,
+				    struct device_driver *drv,
+				    struct device *parent)
 {
 	if (parent)
 		device_lock(parent);
 
 	device_lock(dev);
 	if (!drv || drv == dev->driver)
-		__device_release_driver(dev);
+		__device_release_driver(dev, parent);
 
 	device_unlock(dev);
 	if (parent)
@@ -818,6 +846,10 @@ static void device_release_driver_intern
  *
  * Manually detach device from driver.
  * When called for a USB interface, @dev->parent lock must be held.
+ *
+ * If this function is to be called with @dev->parent lock held, ensure that
+ * the device's consumers are unbound in advance or that their locks can be
+ * acquired under the @dev->parent lock.
  */
 void device_release_driver(struct device *dev)
 {
Index: linux-pm/include/linux/device.h
===================================================================
--- linux-pm.orig/include/linux/device.h
+++ linux-pm/include/linux/device.h
@@ -706,6 +706,35 @@ struct device_dma_parameters {
 	unsigned long segment_boundary_mask;
 };
 
+enum device_link_status {
+	DEVICE_LINK_NO_STATE = -1,
+	DEVICE_LINK_DORMANT = 0,	/* Link not in use. */
+	DEVICE_LINK_AVAILABLE,		/* Supplier driver is present. */
+	DEVICE_LINK_ACTIVE,		/* Consumer driver is present too. */
+	DEVICE_LINK_CONSUMER_PROBE,	/* Consumer is probing. */
+	DEVICE_LINK_SUPPLIER_UNBIND,	/* Supplier is unbinding. */
+};
+
+/*
+ * Device link flags.
+ *
+ * STATELESS: The state machine is not applicable to this link.
+ * AUTOREMOVE: Remove this link automatically on cunsumer driver unbind.
+ */
+#define DEVICE_LINK_STATELESS	(1 << 0)
+#define DEVICE_LINK_AUTOREMOVE	(1 << 1)
+
+struct device_link {
+	struct device *supplier;
+	struct list_head s_node;
+	struct device *consumer;
+	struct list_head c_node;
+	enum device_link_status status;
+	u32 flags;
+	spinlock_t lock;
+	struct rcu_head rcu_head;
+};
+
 /**
  * struct device - The basic device structure
  * @parent:	The device's "parent" device, the device to which it is attached.
@@ -731,6 +760,8 @@ struct device_dma_parameters {
  * 		on.  This shrinks the "Board Support Packages" (BSPs) and
  * 		minimizes board-specific #ifdefs in drivers.
  * @driver_data: Private pointer for driver specific info.
+ * @links_to_consumers: Links to consumer devices.
+ * @links_to_suppliers: Links to supplier devices.
  * @power:	For device power management.
  * 		See Documentation/power/devices.txt for details.
  * @pm_domain:	Provide callbacks that are executed during system suspend,
@@ -797,6 +828,8 @@ struct device {
 					   core doesn't touch it */
 	void		*driver_data;	/* Driver data, set and get with
 					   dev_set/get_drvdata */
+	struct list_head	links_to_consumers;
+	struct list_head	links_to_suppliers;
 	struct dev_pm_info	power;
 	struct dev_pm_domain	*pm_domain;
 
@@ -1113,6 +1146,11 @@ extern void device_shutdown(void);
 /* debugging and troubleshooting/diagnostic helpers. */
 extern const char *dev_driver_string(const struct device *dev);
 
+/* Device links interface. */
+struct device_link *device_link_add(struct device *consumer,
+				    struct device *supplier,
+				    enum device_link_status status, u32 flags);
+void device_link_del(struct device_link *link);
 
 #ifdef CONFIG_PRINTK
 

Re: [RFC/RFT][PATCH v3 2/5] driver core: Functional dependencies tracking support

[Marek Szyprowski]

Hi Rafael,

On 2016-09-16 00:06, Rafael J. Wysocki wrote:
> From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
>
> Currently, there is a problem with taking functional dependencies
> between into account.
>
> What I mean by a "functional dependency" is when the driver of device
> B needs device A to be functional and (generally) its driver to be
> present in order to work properly.  This has certain consequences
> for power management (suspend/resume and runtime PM ordering) and
> shutdown ordering of these devices.  In general, it also implies that
> the driver of A needs to be working for B to be probed successfully
> and it cannot be unbound from the device before the B's driver.
>
> Support for representing those functional dependencies between
> devices is added here to allow the driver core to track them and act
> on them in certain cases where applicable.
>
> The argument for doing that in the driver core is that there are
> quite a few distinct use cases involving device dependencies, they
> are relatively hard to get right in a driver (if one wants to
> address all of them properly) and it only gets worse if multiplied
> by the number of drivers potentially needing to do it.  Morever, at
> least one case (asynchronous system suspend/resume) cannot be handled
> in a single driver at all, because it requires the driver of A to
> wait for B to suspend (during system suspend) and the driver of B to
> wait for A to resume (during system resume).
>
> For this reason, represent dependencies between devices as "links",
> with the help of struct device_link objects each containing pointers
> to the "linked" devices, a list node for each of them, status
> information, flags, a lock and an RCU head for synchronization.
>
> Also add two new list heads, links_to_consumers and links_to_suppliers,
> to struct device to represent the lists of links to the devices that
> depend on the given one (consumers) and to the devices depended on
> by it (suppliers), respectively.
>
> The entire data structure consisting of all of the lists of link
> objects for all devices is protected by SRCU (for list walking)
> and a by mutex (for link object addition/removal).  In addition
> to that, each link object has an internal status field whose
> value reflects what's happening to the devices pointed to by
> the link.  That status field is protected by an internal spinlock.
>
> New links are added by calling device_link_add() which takes four
> arguments: pointers to the devices in question, the initial status
> of the link and flags.  In particular, if DEVICE_LINK_STATELESS is
> set in the flags, the link status is not to be taken into account
> for this link and the driver core will not manage it.  In turn, if
> DEVICE_LINK_AUTOREMOVE is set in the flags, the driver core will
> remove the link automatically when the consumer device driver
> unbinds from it.
>
> One of the actions carried out by device_link_add() is to reorder
> the lists used for device shutdown and system suspend/resume to
> put the consumer device along with all of its children and all of
> its consumers (and so on, recursively) to the ends of those list
> in order to ensure the right ordering between all of the supplier
> and consumer devices.
>
> For this reason, it is not possible to create a link between two
> devices if the would-be supplier device already depends on the
> would-be consumer device as either a direct descendant of it or a
> consumer of one of its direct descendants or one of its consumers
> and so on.
>
> It also is impossible to create a link between a parent and a child
> device (in any direction).
>
> There are two types of link objects, persistent and non-persistent.
> The persistent ones stay around until one of the target devices is
> deleted, while the non-persistent ones are removed automatically when
> the consumer driver unbinds from its device (ie. they are assumed to
> be valid only as long as the consumer device has a driver bound to
> it).  Persistent links are created by default and non-persistent
> links are created when the DEVICE_LINK_AUTOREMOVE flag is passed
> to device_link_add().
>
> Both persistent and non-persistent device links can be deleted
> explicitly with the help of device_link_del().
>
> Links created without the DEVICE_LINK_STATELESS flag set are managed
> by the driver core using a simple state machine.  There are 5 states
> each link can be in: DORMANT (unused), AVAILABLE (the supplier driver
> is present and functional), CONSUMER_PROBE (the consumer driver is
> probing), ACTIVE (both supplier and consumer drivers are present and
> functional), and SUPPLIER_UNBIND (the supplier driver is unbinding).
> The driver core updates the link state automatically depending on
> what happens to the linked devices and for each link state specific
> actions are taken in addition to that.
>
> For example, if the supplier driver unbinds from its device, the
> driver core will also unbind the drivers of all of its consumers
> automatically under the assumption that they cannot function
> properly without the supplier.  Analogously, the driver core will
> only allow the consumer driver to bind to its device is the
> supplier driver is present and functional (ie. the link is in
> the AVAILABLE state).  If that's not the case, it will rely on
> the existing deferred probing mechanism to wait for the supplier
> driver to become available.
>
> Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> ---
>   drivers/base/base.h    |   11 +
>   drivers/base/core.c    |  471 +++++++++++++++++++++++++++++++++++++++++++++++++
>   drivers/base/dd.c      |   42 +++-
>   include/linux/device.h |   38 +++
>   4 files changed, 557 insertions(+), 5 deletions(-)
>
> Index: linux-pm/drivers/base/base.h
> ===================================================================
> --- linux-pm.orig/drivers/base/base.h
> +++ linux-pm/drivers/base/base.h
> @@ -107,6 +107,9 @@ extern void bus_remove_device(struct dev
>   
>   extern int bus_add_driver(struct device_driver *drv);
>   extern void bus_remove_driver(struct device_driver *drv);
> +extern void device_release_driver_internal(struct device *dev,
> +					   struct device_driver *drv,
> +					   struct device *parent);
>   
>   extern void driver_detach(struct device_driver *drv);
>   extern int driver_probe_device(struct device_driver *drv, struct device *dev);
> @@ -152,3 +155,11 @@ extern int devtmpfs_init(void);
>   #else
>   static inline int devtmpfs_init(void) { return 0; }
>   #endif
> +
> +/* Device links */
> +extern int device_links_check_suppliers(struct device *dev);
> +extern void device_links_driver_bound(struct device *dev);
> +extern void device_links_driver_gone(struct device *dev);
> +extern void device_links_no_driver(struct device *dev);
> +extern bool device_links_busy(struct device *dev);
> +extern void device_links_unbind_consumers(struct device *dev);
> Index: linux-pm/drivers/base/core.c
> ===================================================================
> --- linux-pm.orig/drivers/base/core.c
> +++ linux-pm/drivers/base/core.c
> @@ -44,6 +44,450 @@ static int __init sysfs_deprecated_setup
>   early_param("sysfs.deprecated", sysfs_deprecated_setup);
>   #endif
>   
> +/* Device links support. */
> +
> +DEFINE_STATIC_SRCU(device_links_srcu);
> +static DEFINE_MUTEX(device_links_lock);
> +
> +/**
> + * device_is_dependent - Check if one device depends on another one
> + * @dev: Device to check dependencies for.
> + * @target: Device to check against.
> + *
> + * Check if @target depends on @dev or any device dependent on it (its child or
> + * its consumer etc).  Return 1 if that is the case or 0 otherwise.
> + */
> +static int device_is_dependent(struct device *dev, void *target)
> +{
> +	struct device_link *link;
> +	int ret;
> +
> +	if (WARN_ON(dev == target))
> +		return 1;
> +
> +	ret = device_for_each_child(dev, target, device_is_dependent);
> +	if (ret)
> +		return ret;
> +
> +	list_for_each_entry(link, &dev->links_to_consumers, s_node) {
> +		if (WARN_ON(link->consumer == target))
> +			return 1;
> +
> +		ret = device_is_dependent(link->consumer, target);
> +		if (ret)
> +			break;
> +	}
> +	return ret;
> +}
> +
> +static int device_reorder_to_tail(struct device *dev, void *not_used)
> +{
> +	struct device_link *link;
> +
> +	devices_kset_move_last(dev);
> +	device_pm_move_last(dev);
> +	device_for_each_child(dev, NULL, device_reorder_to_tail);
> +	list_for_each_entry(link, &dev->links_to_consumers, s_node)
> +		device_reorder_to_tail(link->consumer, NULL);
> +
> +	return 0;
> +}
> +
> +/**
> + * device_link_add - Create a link between two devices.
> + * @consumer: Consumer end of the link.
> + * @supplier: Supplier end of the link.
> + * @status: The initial status of the link.
> + * @flags: Link flags.
> + *
> + * If the DEVICE_LINK_STATELESS flag is set, @status is ignored.  Otherwise,
> + * the caller is responsible for ensuring that @status reflects the current
> + * status of both @consumer and @supplier.
> + *
> + * If the DEVICE_LINK_AUTOREMOVE is set, the link will be removed automatically
> + * when the consumer device driver unbinds from it.  The combination of both
> + * DEVICE_LINK_AUTOREMOVE and DEVICE_LINK_STATELESS set is invalid and will
> + * cause NULL to be returned.
> + *
> + * A side effect of the link creation is re-ordering of dpm_list and the
> + * devices_kset list by moving the consumer device and all devices depending
> + * on it to the ends of these lists.
> + */
> +struct device_link *device_link_add(struct device *consumer,
> +				    struct device *supplier,
> +				    enum device_link_status status, u32 flags)
> +{
> +	struct device_link *link;
> +
> +	if (!consumer || !supplier || supplier == consumer->parent ||
> +	    ((flags & DEVICE_LINK_STATELESS) && (flags & DEVICE_LINK_AUTOREMOVE)))
> +		return NULL;
> +
> +	mutex_lock(&device_links_lock);
> +
> +	/*
> +	 * If there is a reverse dependency between the consumer and the
> +	 * supplier already in the graph, return NULL.
> +	 */
> +	if (device_is_dependent(consumer, supplier)) {
> +		link = NULL;
> +		goto out;
> +	}
> +
> +	list_for_each_entry(link, &supplier->links_to_consumers, s_node)
> +		if (link->consumer == consumer)
> +			goto out;
> +
> +	link = kmalloc(sizeof(*link), GFP_KERNEL);
> +	if (!link)
> +		goto out;
> +
> +	get_device(supplier);
> +	link->supplier = supplier;
> +	INIT_LIST_HEAD(&link->s_node);
> +	get_device(consumer);
> +	link->consumer = consumer;
> +	INIT_LIST_HEAD(&link->c_node);
> +	spin_lock_init(&link->lock);
> +	link->flags = flags;
> +	link->status = (flags & DEVICE_LINK_STATELESS) ?
> +					DEVICE_LINK_NO_STATE : status;
> +
> +	/*
> +	 * Move the consumer and all of the devices depending on it to the end
> +	 * of dpm_list and the devices_kset list.
> +	 *
> +	 * It is necessary to hold dpm_list locked throughout all that or else
> +	 * we may end up suspending with a wrong ordering of it.
> +	 */
> +	device_pm_lock();
> +	device_reorder_to_tail(consumer, NULL);
> +	device_pm_unlock();
> +
> +	list_add_tail_rcu(&link->s_node, &supplier->links_to_consumers);
> +	list_add_tail_rcu(&link->c_node, &consumer->links_to_suppliers);
> +
> +	dev_info(consumer, "Linked as a consumer to %s\n", dev_name(supplier));
> +
> + out:
> +	mutex_unlock(&device_links_lock);
> +	return link;
> +}
> +EXPORT_SYMBOL_GPL(device_link_add);
> +
> +static void __device_link_free_srcu(struct rcu_head *rhead)
> +{
> +	struct device_link *link;
> +
> +	link = container_of(rhead, struct device_link, rcu_head);
> +	put_device(link->consumer);
> +	put_device(link->supplier);
> +	kfree(link);
> +}
> +
> +static void __device_link_del(struct device_link *link)
> +{
> +	dev_info(link->consumer, "Dropping the link to %s\n",
> +		 dev_name(link->supplier));
> +
> +	list_del_rcu(&link->s_node);
> +	list_del_rcu(&link->c_node);
> +	call_srcu(&device_links_srcu, &link->rcu_head, __device_link_free_srcu);
> +}
> +
> +/**
> + * device_link_del - Delete a link between two devices.
> + * @link: Device link to delete.
> + *
> + * The caller must ensure proper synchronization of this function with runtime
> + * PM.
> + */
> +void device_link_del(struct device_link *link)
> +{
> +	mutex_lock(&device_links_lock);
> +	device_pm_lock();
> +	__device_link_del(link);
> +	device_pm_unlock();
> +	mutex_unlock(&device_links_lock);
> +}
> +EXPORT_SYMBOL_GPL(device_link_del);
> +
> +static int device_links_read_lock(void)
> +{
> +	return srcu_read_lock(&device_links_srcu);
> +}
> +
> +static void device_links_read_unlock(int idx)
> +{
> +	return srcu_read_unlock(&device_links_srcu, idx);
> +}
> +
> +static void device_links_missing_supplier(struct device *dev)
> +{
> +	struct device_link *link;
> +
> +	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
> +		spin_lock(&link->lock);
> +
> +		if (link->status == DEVICE_LINK_CONSUMER_PROBE)
> +			link->status = DEVICE_LINK_AVAILABLE;
> +
> +		spin_unlock(&link->lock);
> +	}
> +}
> +
> +/**
> + * device_links_check_suppliers - Check supplier devices for this one.
> + * @dev: Consumer device.
> + *
> + * Check links from this device to any suppliers.  Walk the list of the device's
> + * consumer links and see if all of the suppliers are available.  If not, simply
> + * return -EPROBE_DEFER.
> + *
> + * Walk the list under SRCU and check each link's status field under its lock.
> + *
> + * We need to guarantee that the supplier will not go away after the check has
> + * been positive here.  It only can go away in __device_release_driver() and
> + * that function  checks the device's links to consumers.  This means we need to
> + * mark the link as "consumer probe in progress" to make the supplier removal
> + * wait for us to complete (or bad things may happen).
> + *
> + * Links with the DEVICE_LINK_STATELESS flag set are ignored.
> + */
> +int device_links_check_suppliers(struct device *dev)
> +{
> +	struct device_link *link;
> +	int idx, ret = 0;
> +
> +	idx = device_links_read_lock();
> +
> +	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
> +		if (link->flags & DEVICE_LINK_STATELESS)
> +			continue;
> +
> +		spin_lock(&link->lock);
> +		if (link->status != DEVICE_LINK_AVAILABLE) {
> +			spin_unlock(&link->lock);
> +			device_links_missing_supplier(dev);
> +			ret = -EPROBE_DEFER;
> +			break;
> +		}
> +		link->status = DEVICE_LINK_CONSUMER_PROBE;
> +		spin_unlock(&link->lock);
> +	}
> +
> +	device_links_read_unlock(idx);
> +	return ret;
> +}
> +
> +/**
> + * device_links_driver_bound - Update device links after probing its driver.
> + * @dev: Device to update the links for.
> + *
> + * The probe has been successful, so update links from this device to any
> + * consumers by changing their status to "available".
> + *
> + * Also change the status of @dev's links to suppliers to "active".
> + *
> + * Links with the DEVICE_LINK_STATELESS flag set are ignored.
> + */
> +void device_links_driver_bound(struct device *dev)
> +{
> +	struct device_link *link;
> +	int idx;
> +
> +	idx = device_links_read_lock();
> +
> +	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
> +		if (link->flags & DEVICE_LINK_STATELESS)
> +			continue;
> +
> +		spin_lock(&link->lock);
> +		WARN_ON(link->status != DEVICE_LINK_DORMANT);
> +		link->status = DEVICE_LINK_AVAILABLE;
> +		spin_unlock(&link->lock);
> +	}
> +
> +	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
> +		if (link->flags & DEVICE_LINK_STATELESS)
> +			continue;
> +
> +		spin_lock(&link->lock);
> +		WARN_ON(link->status != DEVICE_LINK_CONSUMER_PROBE);
> +		link->status = DEVICE_LINK_ACTIVE;
> +		spin_unlock(&link->lock);
> +	}
> +
> +	device_links_read_unlock(idx);
> +}
> +
> +/**
> + * device_links_driver_gone - Update links after driver removal.
> + * @dev: Device whose driver has gone away.
> + *
> + * Update links to consumers for @dev by changing their status to "dormant".
> + *
> + * Links with the DEVICE_LINK_STATELESS flag set are ignored.
> + */
> +void device_links_driver_gone(struct device *dev)
> +{
> +	struct device_link *link;
> +	int idx;
> +
> +	idx = device_links_read_lock();
> +
> +	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
> +		if (link->flags & DEVICE_LINK_STATELESS)
> +			continue;
> +
> +		WARN_ON(link->flags & DEVICE_LINK_AUTOREMOVE);
> +		spin_lock(&link->lock);
> +		WARN_ON(link->status != DEVICE_LINK_SUPPLIER_UNBIND);
> +		link->status = DEVICE_LINK_DORMANT;
> +		spin_unlock(&link->lock);
> +	}
> +
> +	device_links_read_unlock(idx);
> +}
> +
> +/**
> + * device_links_no_driver - Update links of a device without a driver.
> + * @dev: Device without a drvier.
> + *
> + * Delete all non-persistent links from this device to any suppliers.
> + *
> + * Persistent links stay around, but their status is changed to "available",
> + * unless they already are in the "supplier unbind in progress" state in which
> + * case they need not be updated.
> + *
> + * Links with the DEVICE_LINK_STATELESS flag set are ignored.
> + */
> +void device_links_no_driver(struct device *dev)
> +{
> +	struct device_link *link, *ln;
> +
> +	mutex_lock(&device_links_lock);
> +
> +	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_suppliers, c_node)

missing "{"

> +		if (link->flags & DEVICE_LINK_STATELESS)
> +			continue;
> +
> +		if (link->flags & DEVICE_LINK_AUTOREMOVE) {
> +			__device_link_del(link);
> +		} else {
> +			spin_lock(&link->lock);
> +
> +			if (link->status != DEVICE_LINK_SUPPLIER_UNBIND)
> +				link->status = DEVICE_LINK_AVAILABLE;
> +
> +			spin_unlock(&link->lock);
> +		}

missing "}"

> +
> +	mutex_unlock(&device_links_lock);
> +}
> +
> +/**
> + * device_links_busy - Check if there are any busy links to consumers.
> + * @dev: Device to check.
> + *
> + * Check each consumer of the device and return 'true' if its link's status
> + * is one of "consumer probe" or "active" (meaning that the given consumer is
> + * probing right now or its driver is present).  Otherwise, change the link
> + * state to "supplier unbind" to prevent the consumer from being probed
> + * successfully going forward.
> + *
> + * Return 'false' if there are no probing or active consumers.
> + *
> + * Links with the DEVICE_LINK_STATELESS flag set are ignored.
> + */
> +bool device_links_busy(struct device *dev)
> +{
> +	struct device_link *link;
> +	int idx;
> +	bool ret = false;
> +
> +	idx = device_links_read_lock();
> +
> +	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
> +		if (link->flags & DEVICE_LINK_STATELESS)
> +			continue;
> +
> +		spin_lock(&link->lock);
> +		if (link->status == DEVICE_LINK_CONSUMER_PROBE
> +		    || link->status == DEVICE_LINK_ACTIVE) {
> +			spin_unlock(&link->lock);
> +			ret = true;
> +			break;
> +		}
> +		link->status = DEVICE_LINK_SUPPLIER_UNBIND;
> +		spin_unlock(&link->lock);
> +	}
> +
> +	device_links_read_unlock(idx);
> +	return ret;
> +}
> +
> +/**
> + * device_links_unbind_consumers - Force unbind consumers of the given device.
> + * @dev: Device to unbind the consumers of.
> + *
> + * Walk the list of links to consumers for @dev and if any of them is in the
> + * "consumer probe" state, wait for all device probes in progress to complete
> + * and start over.
> + *
> + * If that's not the case, change the status of the link to "supplier unbind"
> + * and check if the link was in the "active" state.  If so, force the consumer
> + * driver to unbind and start over (the consumer will not re-probe as we have
> + * changed the state of the link already).
> + *
> + * Links with the DEVICE_LINK_STATELESS flag set are ignored.
> + */
> +void device_links_unbind_consumers(struct device *dev)
> +{
> +	struct device_link *link;
> +	int idx;
> +
> + start:
> +	idx = device_links_read_lock();
> +
> +	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
> +		enum device_link_status status;
> +
> +		if (link->flags & DEVICE_LINK_STATELESS)
> +			continue;
> +
> +		spin_lock(&link->lock);
> +		status = link->status;
> +		if (status == DEVICE_LINK_CONSUMER_PROBE) {
> +			spin_unlock(&link->lock);
> +
> +			device_links_read_unlock(idx);
> +
> +			wait_for_device_probe();
> +			goto start;
> +		}
> +		link->status = DEVICE_LINK_SUPPLIER_UNBIND;
> +		if (status == DEVICE_LINK_ACTIVE) {
> +			struct device *consumer = link->consumer;
> +
> +			get_device(consumer);
> +			spin_unlock(&link->lock);
> +
> +			device_links_read_unlock(idx);
> +
> +			device_release_driver_internal(consumer, NULL,
> +						       consumer->parent);
> +			put_device(consumer);
> +			goto start;
> +		}
> +		spin_unlock(&link->lock);
> +	}
> +
> +	device_links_read_unlock(idx);
> +}
> +
> +/* Device links support end. */
> +
>   int (*platform_notify)(struct device *dev) = NULL;
>   int (*platform_notify_remove)(struct device *dev) = NULL;
>   static struct kobject *dev_kobj;
> @@ -711,6 +1155,8 @@ void device_initialize(struct device *de
>   #ifdef CONFIG_GENERIC_MSI_IRQ
>   	INIT_LIST_HEAD(&dev->msi_list);
>   #endif
> +	INIT_LIST_HEAD(&dev->links_to_consumers);
> +	INIT_LIST_HEAD(&dev->links_to_suppliers);
>   }
>   EXPORT_SYMBOL_GPL(device_initialize);
>   
> @@ -1233,6 +1679,7 @@ void device_del(struct device *dev)
>   {
>   	struct device *parent = dev->parent;
>   	struct class_interface *class_intf;
> +	struct device_link *link, *ln;
>   
>   	/* Notify clients of device removal.  This call must come
>   	 * before dpm_sysfs_remove().
> @@ -1240,6 +1687,30 @@ void device_del(struct device *dev)
>   	if (dev->bus)
>   		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
>   					     BUS_NOTIFY_DEL_DEVICE, dev);
> +
> +	/*
> +	 * Delete all of the remaining links from this device to any other
> +	 * devices (either consumers or suppliers).
> +	 *
> +	 * This requires that all links be dormant, so warn if that's no the
> +	 * case.
> +	 */
> +	mutex_lock(&device_links_lock);
> +
> +	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_suppliers, c_node) {
> +		WARN_ON(link->status != DEVICE_LINK_DORMANT &&
> +			!(link->flags & DEVICE_LINK_STATELESS));
> +		__device_link_del(link);
> +	}
> +
> +	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_consumers, s_node) {
> +		WARN_ON(link->status != DEVICE_LINK_DORMANT &&
> +			!(link->flags & DEVICE_LINK_STATELESS));
> +		__device_link_del(link);
> +	}
> +
> +	mutex_unlock(&device_links_lock);
> +
>   	dpm_sysfs_remove(dev);
>   	if (parent)
>   		klist_del(&dev->p->knode_parent);
> Index: linux-pm/drivers/base/dd.c
> ===================================================================
> --- linux-pm.orig/drivers/base/dd.c
> +++ linux-pm/drivers/base/dd.c
> @@ -249,6 +249,7 @@ static void driver_bound(struct device *
>   		 __func__, dev_name(dev));
>   
>   	klist_add_tail(&dev->p->knode_driver, &dev->driver->p->klist_devices);
> +	device_links_driver_bound(dev);
>   
>   	device_pm_check_callbacks(dev);
>   
> @@ -399,6 +400,7 @@ probe_failed:
>   		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
>   					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
>   pinctrl_bind_failed:
> +	device_links_no_driver(dev);
>   	devres_release_all(dev);
>   	driver_sysfs_remove(dev);
>   	dev->driver = NULL;
> @@ -489,6 +491,10 @@ int driver_probe_device(struct device_dr
>   	if (!device_is_registered(dev))
>   		return -ENODEV;
>   
> +	ret = device_links_check_suppliers(dev);
> +	if (ret)
> +		return ret;
> +
>   	pr_debug("bus: '%s': %s: matched device %s with driver %s\n",
>   		 drv->bus->name, __func__, dev_name(dev), drv->name);
>   
> @@ -756,7 +762,7 @@ EXPORT_SYMBOL_GPL(driver_attach);
>    * __device_release_driver() must be called with @dev lock held.
>    * When called for a USB interface, @dev->parent lock must be held as well.
>    */
> -static void __device_release_driver(struct device *dev)
> +static void __device_release_driver(struct device *dev, struct device *parent)
>   {
>   	struct device_driver *drv;
>   
> @@ -765,6 +771,25 @@ static void __device_release_driver(stru
>   		if (driver_allows_async_probing(drv))
>   			async_synchronize_full();
>   
> +		while (device_links_busy(dev)) {
> +			device_unlock(dev);
> +			if (parent)
> +				device_unlock(parent);
> +
> +			device_links_unbind_consumers(dev);
> +			if (parent)
> +				device_lock(parent);
> +
> +			device_lock(dev);
> +			/*
> +			 * A concurrent invocation of the same function might
> +			 * have released the driver successfully while this one
> +			 * was waiting, so check for that.
> +			 */
> +			if (dev->driver != drv)
> +				return;
> +		}
> +
>   		pm_runtime_get_sync(dev);
>   
>   		driver_sysfs_remove(dev);
> @@ -780,6 +805,9 @@ static void __device_release_driver(stru
>   			dev->bus->remove(dev);
>   		else if (drv->remove)
>   			drv->remove(dev);
> +
> +		device_links_driver_gone(dev);
> +		device_links_no_driver(dev);
>   		devres_release_all(dev);
>   		dev->driver = NULL;
>   		dev_set_drvdata(dev, NULL);
> @@ -796,16 +824,16 @@ static void __device_release_driver(stru
>   	}
>   }
>   
> -static void device_release_driver_internal(struct device *dev,
> -					   struct device_driver *drv,
> -					   struct device *parent)
> +void device_release_driver_internal(struct device *dev,
> +				    struct device_driver *drv,
> +				    struct device *parent)
>   {
>   	if (parent)
>   		device_lock(parent);
>   
>   	device_lock(dev);
>   	if (!drv || drv == dev->driver)
> -		__device_release_driver(dev);
> +		__device_release_driver(dev, parent);
>   
>   	device_unlock(dev);
>   	if (parent)
> @@ -818,6 +846,10 @@ static void device_release_driver_intern
>    *
>    * Manually detach device from driver.
>    * When called for a USB interface, @dev->parent lock must be held.
> + *
> + * If this function is to be called with @dev->parent lock held, ensure that
> + * the device's consumers are unbound in advance or that their locks can be
> + * acquired under the @dev->parent lock.
>    */
>   void device_release_driver(struct device *dev)
>   {
> Index: linux-pm/include/linux/device.h
> ===================================================================
> --- linux-pm.orig/include/linux/device.h
> +++ linux-pm/include/linux/device.h
> @@ -706,6 +706,35 @@ struct device_dma_parameters {
>   	unsigned long segment_boundary_mask;
>   };
>   
> +enum device_link_status {
> +	DEVICE_LINK_NO_STATE = -1,
> +	DEVICE_LINK_DORMANT = 0,	/* Link not in use. */
> +	DEVICE_LINK_AVAILABLE,		/* Supplier driver is present. */
> +	DEVICE_LINK_ACTIVE,		/* Consumer driver is present too. */
> +	DEVICE_LINK_CONSUMER_PROBE,	/* Consumer is probing. */
> +	DEVICE_LINK_SUPPLIER_UNBIND,	/* Supplier is unbinding. */
> +};
> +
> +/*
> + * Device link flags.
> + *
> + * STATELESS: The state machine is not applicable to this link.
> + * AUTOREMOVE: Remove this link automatically on cunsumer driver unbind.
> + */
> +#define DEVICE_LINK_STATELESS	(1 << 0)
> +#define DEVICE_LINK_AUTOREMOVE	(1 << 1)
> +
> +struct device_link {
> +	struct device *supplier;
> +	struct list_head s_node;
> +	struct device *consumer;
> +	struct list_head c_node;
> +	enum device_link_status status;
> +	u32 flags;
> +	spinlock_t lock;
> +	struct rcu_head rcu_head;
> +};
> +
>   /**
>    * struct device - The basic device structure
>    * @parent:	The device's "parent" device, the device to which it is attached.
> @@ -731,6 +760,8 @@ struct device_dma_parameters {
>    * 		on.  This shrinks the "Board Support Packages" (BSPs) and
>    * 		minimizes board-specific #ifdefs in drivers.
>    * @driver_data: Private pointer for driver specific info.
> + * @links_to_consumers: Links to consumer devices.
> + * @links_to_suppliers: Links to supplier devices.
>    * @power:	For device power management.
>    * 		See Documentation/power/devices.txt for details.
>    * @pm_domain:	Provide callbacks that are executed during system suspend,
> @@ -797,6 +828,8 @@ struct device {
>   					   core doesn't touch it */
>   	void		*driver_data;	/* Driver data, set and get with
>   					   dev_set/get_drvdata */
> +	struct list_head	links_to_consumers;
> +	struct list_head	links_to_suppliers;
>   	struct dev_pm_info	power;
>   	struct dev_pm_domain	*pm_domain;
>   
> @@ -1113,6 +1146,11 @@ extern void device_shutdown(void);
>   /* debugging and troubleshooting/diagnostic helpers. */
>   extern const char *dev_driver_string(const struct device *dev);
>   
> +/* Device links interface. */
> +struct device_link *device_link_add(struct device *consumer,
> +				    struct device *supplier,
> +				    enum device_link_status status, u32 flags);
> +void device_link_del(struct device_link *link);
>   
>   #ifdef CONFIG_PRINTK
>   
>
>
>

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [RFC/RFT][PATCH v3 2/5] driver core: Functional dependencies tracking support

[Rafael J. Wysocki]

On Fri, Sep 16, 2016 at 9:53 AM, Marek Szyprowski
<m.szyprowski@samsung.com> wrote:
> Hi Rafael,
>
>
> On 2016-09-16 00:06, Rafael J. Wysocki wrote:
>>
>> From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
>>

[cut]

>> +/**
>> + * device_links_no_driver - Update links of a device without a driver.
>> + * @dev: Device without a drvier.
>> + *
>> + * Delete all non-persistent links from this device to any suppliers.
>> + *
>> + * Persistent links stay around, but their status is changed to
>> "available",
>> + * unless they already are in the "supplier unbind in progress" state in
>> which
>> + * case they need not be updated.
>> + *
>> + * Links with the DEVICE_LINK_STATELESS flag set are ignored.
>> + */
>> +void device_links_no_driver(struct device *dev)
>> +{
>> +       struct device_link *link, *ln;
>> +
>> +       mutex_lock(&device_links_lock);
>> +
>> +       list_for_each_entry_safe_reverse(link, ln,
>> &dev->links_to_suppliers, c_node)
>
>
> missing "{"
>
>> +               if (link->flags & DEVICE_LINK_STATELESS)
>> +                       continue;
>> +
>> +               if (link->flags & DEVICE_LINK_AUTOREMOVE) {
>> +                       __device_link_del(link);
>> +               } else {
>> +                       spin_lock(&link->lock);
>> +
>> +                       if (link->status != DEVICE_LINK_SUPPLIER_UNBIND)
>> +                               link->status = DEVICE_LINK_AVAILABLE;
>> +
>> +                       spin_unlock(&link->lock);
>> +               }
>
>
> missing "}"
>

Sorry for the breakage and thanks for the fix!

Rafael

[Update][RFC/RFT][PATCH v3 2/5] driver core: Functional dependencies tracking support

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Currently, there is a problem with taking functional dependencies
between into account.

What I mean by a "functional dependency" is when the driver of device
B needs device A to be functional and (generally) its driver to be
present in order to work properly.  This has certain consequences
for power management (suspend/resume and runtime PM ordering) and
shutdown ordering of these devices.  In general, it also implies that
the driver of A needs to be working for B to be probed successfully
and it cannot be unbound from the device before the B's driver.  

Support for representing those functional dependencies between
devices is added here to allow the driver core to track them and act
on them in certain cases where applicable.

The argument for doing that in the driver core is that there are
quite a few distinct use cases involving device dependencies, they
are relatively hard to get right in a driver (if one wants to
address all of them properly) and it only gets worse if multiplied
by the number of drivers potentially needing to do it.  Morever, at
least one case (asynchronous system suspend/resume) cannot be handled
in a single driver at all, because it requires the driver of A to
wait for B to suspend (during system suspend) and the driver of B to
wait for A to resume (during system resume).

For this reason, represent dependencies between devices as "links",
with the help of struct device_link objects each containing pointers
to the "linked" devices, a list node for each of them, status
information, flags, a lock and an RCU head for synchronization.

Also add two new list heads, links_to_consumers and links_to_suppliers,
to struct device to represent the lists of links to the devices that
depend on the given one (consumers) and to the devices depended on
by it (suppliers), respectively.

The entire data structure consisting of all of the lists of link
objects for all devices is protected by SRCU (for list walking)
and a by mutex (for link object addition/removal).  In addition
to that, each link object has an internal status field whose
value reflects what's happening to the devices pointed to by
the link.  That status field is protected by an internal spinlock.

New links are added by calling device_link_add() which takes four
arguments: pointers to the devices in question, the initial status
of the link and flags.  In particular, if DEVICE_LINK_STATELESS is
set in the flags, the link status is not to be taken into account
for this link and the driver core will not manage it.  In turn, if
DEVICE_LINK_AUTOREMOVE is set in the flags, the driver core will
remove the link automatically when the consumer device driver
unbinds from it.

One of the actions carried out by device_link_add() is to reorder
the lists used for device shutdown and system suspend/resume to
put the consumer device along with all of its children and all of
its consumers (and so on, recursively) to the ends of those list
in order to ensure the right ordering between all of the supplier
and consumer devices.

For this reason, it is not possible to create a link between two
devices if the would-be supplier device already depends on the
would-be consumer device as either a direct descendant of it or a
consumer of one of its direct descendants or one of its consumers
and so on.

It also is impossible to create a link between a parent and a child
device (in any direction).

There are two types of link objects, persistent and non-persistent.
The persistent ones stay around until one of the target devices is
deleted, while the non-persistent ones are removed automatically when
the consumer driver unbinds from its device (ie. they are assumed to
be valid only as long as the consumer device has a driver bound to
it).  Persistent links are created by default and non-persistent
links are created when the DEVICE_LINK_AUTOREMOVE flag is passed
to device_link_add().

Both persistent and non-persistent device links can be deleted
explicitly with the help of device_link_del().

Links created without the DEVICE_LINK_STATELESS flag set are managed
by the driver core using a simple state machine.  There are 5 states
each link can be in: DORMANT (unused), AVAILABLE (the supplier driver
is present and functional), CONSUMER_PROBE (the consumer driver is
probing), ACTIVE (both supplier and consumer drivers are present and
functional), and SUPPLIER_UNBIND (the supplier driver is unbinding).
The driver core updates the link state automatically depending on
what happens to the linked devices and for each link state specific
actions are taken in addition to that.

For example, if the supplier driver unbinds from its device, the
driver core will also unbind the drivers of all of its consumers
automatically under the assumption that they cannot function
properly without the supplier.  Analogously, the driver core will
only allow the consumer driver to bind to its device is the
supplier driver is present and functional (ie. the link is in
the AVAILABLE state).  If that's not the case, it will rely on
the existing deferred probing mechanism to wait for the supplier
driver to become available.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
---

The update adds missing braces in device_links_no_driver() found by Marek.

---
 drivers/base/base.h    |   11 +
 drivers/base/core.c    |  472 +++++++++++++++++++++++++++++++++++++++++++++++++
 drivers/base/dd.c      |   42 +++-
 include/linux/device.h |   38 +++
 4 files changed, 558 insertions(+), 5 deletions(-)

Index: linux-pm/drivers/base/base.h
===================================================================
--- linux-pm.orig/drivers/base/base.h
+++ linux-pm/drivers/base/base.h
@@ -107,6 +107,9 @@ extern void bus_remove_device(struct dev
 
 extern int bus_add_driver(struct device_driver *drv);
 extern void bus_remove_driver(struct device_driver *drv);
+extern void device_release_driver_internal(struct device *dev,
+					   struct device_driver *drv,
+					   struct device *parent);
 
 extern void driver_detach(struct device_driver *drv);
 extern int driver_probe_device(struct device_driver *drv, struct device *dev);
@@ -152,3 +155,11 @@ extern int devtmpfs_init(void);
 #else
 static inline int devtmpfs_init(void) { return 0; }
 #endif
+
+/* Device links */
+extern int device_links_check_suppliers(struct device *dev);
+extern void device_links_driver_bound(struct device *dev);
+extern void device_links_driver_gone(struct device *dev);
+extern void device_links_no_driver(struct device *dev);
+extern bool device_links_busy(struct device *dev);
+extern void device_links_unbind_consumers(struct device *dev);
Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -44,6 +44,451 @@ static int __init sysfs_deprecated_setup
 early_param("sysfs.deprecated", sysfs_deprecated_setup);
 #endif
 
+/* Device links support. */
+
+DEFINE_STATIC_SRCU(device_links_srcu);
+static DEFINE_MUTEX(device_links_lock);
+
+/**
+ * device_is_dependent - Check if one device depends on another one
+ * @dev: Device to check dependencies for.
+ * @target: Device to check against.
+ *
+ * Check if @target depends on @dev or any device dependent on it (its child or
+ * its consumer etc).  Return 1 if that is the case or 0 otherwise.
+ */
+static int device_is_dependent(struct device *dev, void *target)
+{
+	struct device_link *link;
+	int ret;
+
+	if (WARN_ON(dev == target))
+		return 1;
+
+	ret = device_for_each_child(dev, target, device_is_dependent);
+	if (ret)
+		return ret;
+
+	list_for_each_entry(link, &dev->links_to_consumers, s_node) {
+		if (WARN_ON(link->consumer == target))
+			return 1;
+
+		ret = device_is_dependent(link->consumer, target);
+		if (ret)
+			break;
+	}
+	return ret;
+}
+
+static int device_reorder_to_tail(struct device *dev, void *not_used)
+{
+	struct device_link *link;
+
+	devices_kset_move_last(dev);
+	device_pm_move_last(dev);
+	device_for_each_child(dev, NULL, device_reorder_to_tail);
+	list_for_each_entry(link, &dev->links_to_consumers, s_node)
+		device_reorder_to_tail(link->consumer, NULL);
+
+	return 0;
+}
+
+/**
+ * device_link_add - Create a link between two devices.
+ * @consumer: Consumer end of the link.
+ * @supplier: Supplier end of the link.
+ * @status: The initial status of the link.
+ * @flags: Link flags.
+ *
+ * If the DEVICE_LINK_STATELESS flag is set, @status is ignored.  Otherwise,
+ * the caller is responsible for ensuring that @status reflects the current
+ * status of both @consumer and @supplier.
+ *
+ * If the DEVICE_LINK_AUTOREMOVE is set, the link will be removed automatically
+ * when the consumer device driver unbinds from it.  The combination of both
+ * DEVICE_LINK_AUTOREMOVE and DEVICE_LINK_STATELESS set is invalid and will
+ * cause NULL to be returned.
+ *
+ * A side effect of the link creation is re-ordering of dpm_list and the
+ * devices_kset list by moving the consumer device and all devices depending
+ * on it to the ends of these lists.
+ */
+struct device_link *device_link_add(struct device *consumer,
+				    struct device *supplier,
+				    enum device_link_status status, u32 flags)
+{
+	struct device_link *link;
+
+	if (!consumer || !supplier || supplier == consumer->parent ||
+	    ((flags & DEVICE_LINK_STATELESS) && (flags & DEVICE_LINK_AUTOREMOVE)))
+		return NULL;
+
+	mutex_lock(&device_links_lock);
+
+	/*
+	 * If there is a reverse dependency between the consumer and the
+	 * supplier already in the graph, return NULL.
+	 */
+	if (device_is_dependent(consumer, supplier)) {
+		link = NULL;
+		goto out;
+	}
+
+	list_for_each_entry(link, &supplier->links_to_consumers, s_node)
+		if (link->consumer == consumer)
+			goto out;
+
+	link = kmalloc(sizeof(*link), GFP_KERNEL);
+	if (!link)
+		goto out;
+
+	get_device(supplier);
+	link->supplier = supplier;
+	INIT_LIST_HEAD(&link->s_node);
+	get_device(consumer);
+	link->consumer = consumer;
+	INIT_LIST_HEAD(&link->c_node);
+	spin_lock_init(&link->lock);
+	link->flags = flags;
+	link->status = (flags & DEVICE_LINK_STATELESS) ?
+					DEVICE_LINK_NO_STATE : status;
+
+	/*
+	 * Move the consumer and all of the devices depending on it to the end
+	 * of dpm_list and the devices_kset list.
+	 *
+	 * It is necessary to hold dpm_list locked throughout all that or else
+	 * we may end up suspending with a wrong ordering of it.
+	 */
+	device_pm_lock();
+	device_reorder_to_tail(consumer, NULL);
+	device_pm_unlock();
+
+	list_add_tail_rcu(&link->s_node, &supplier->links_to_consumers);
+	list_add_tail_rcu(&link->c_node, &consumer->links_to_suppliers);
+
+	dev_info(consumer, "Linked as a consumer to %s\n", dev_name(supplier));
+
+ out:
+	mutex_unlock(&device_links_lock);
+	return link;
+}
+EXPORT_SYMBOL_GPL(device_link_add);
+
+static void __device_link_free_srcu(struct rcu_head *rhead)
+{
+	struct device_link *link;
+
+	link = container_of(rhead, struct device_link, rcu_head);
+	put_device(link->consumer);
+	put_device(link->supplier);
+	kfree(link);
+}
+
+static void __device_link_del(struct device_link *link)
+{
+	dev_info(link->consumer, "Dropping the link to %s\n",
+		 dev_name(link->supplier));
+
+	list_del_rcu(&link->s_node);
+	list_del_rcu(&link->c_node);
+	call_srcu(&device_links_srcu, &link->rcu_head, __device_link_free_srcu);
+}
+
+/**
+ * device_link_del - Delete a link between two devices.
+ * @link: Device link to delete.
+ *
+ * The caller must ensure proper synchronization of this function with runtime
+ * PM.
+ */
+void device_link_del(struct device_link *link)
+{
+	mutex_lock(&device_links_lock);
+	device_pm_lock();
+	__device_link_del(link);
+	device_pm_unlock();
+	mutex_unlock(&device_links_lock);
+}
+EXPORT_SYMBOL_GPL(device_link_del);
+
+static int device_links_read_lock(void)
+{
+	return srcu_read_lock(&device_links_srcu);
+}
+
+static void device_links_read_unlock(int idx)
+{
+	return srcu_read_unlock(&device_links_srcu, idx);
+}
+
+static void device_links_missing_supplier(struct device *dev)
+{
+	struct device_link *link;
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
+		spin_lock(&link->lock);
+
+		if (link->status == DEVICE_LINK_CONSUMER_PROBE)
+			link->status = DEVICE_LINK_AVAILABLE;
+
+		spin_unlock(&link->lock);
+	}
+}
+
+/**
+ * device_links_check_suppliers - Check supplier devices for this one.
+ * @dev: Consumer device.
+ *
+ * Check links from this device to any suppliers.  Walk the list of the device's
+ * consumer links and see if all of the suppliers are available.  If not, simply
+ * return -EPROBE_DEFER.
+ *
+ * Walk the list under SRCU and check each link's status field under its lock.
+ *
+ * We need to guarantee that the supplier will not go away after the check has
+ * been positive here.  It only can go away in __device_release_driver() and
+ * that function  checks the device's links to consumers.  This means we need to
+ * mark the link as "consumer probe in progress" to make the supplier removal
+ * wait for us to complete (or bad things may happen).
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ */
+int device_links_check_suppliers(struct device *dev)
+{
+	struct device_link *link;
+	int idx, ret = 0;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		spin_lock(&link->lock);
+		if (link->status != DEVICE_LINK_AVAILABLE) {
+			spin_unlock(&link->lock);
+			device_links_missing_supplier(dev);
+			ret = -EPROBE_DEFER;
+			break;
+		}
+		link->status = DEVICE_LINK_CONSUMER_PROBE;
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+	return ret;
+}
+
+/**
+ * device_links_driver_bound - Update device links after probing its driver.
+ * @dev: Device to update the links for.
+ *
+ * The probe has been successful, so update links from this device to any
+ * consumers by changing their status to "available".
+ *
+ * Also change the status of @dev's links to suppliers to "active".
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ */
+void device_links_driver_bound(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		spin_lock(&link->lock);
+		WARN_ON(link->status != DEVICE_LINK_DORMANT);
+		link->status = DEVICE_LINK_AVAILABLE;
+		spin_unlock(&link->lock);
+	}
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		spin_lock(&link->lock);
+		WARN_ON(link->status != DEVICE_LINK_CONSUMER_PROBE);
+		link->status = DEVICE_LINK_ACTIVE;
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+}
+
+/**
+ * device_links_driver_gone - Update links after driver removal.
+ * @dev: Device whose driver has gone away.
+ *
+ * Update links to consumers for @dev by changing their status to "dormant".
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ */
+void device_links_driver_gone(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		WARN_ON(link->flags & DEVICE_LINK_AUTOREMOVE);
+		spin_lock(&link->lock);
+		WARN_ON(link->status != DEVICE_LINK_SUPPLIER_UNBIND);
+		link->status = DEVICE_LINK_DORMANT;
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+}
+
+/**
+ * device_links_no_driver - Update links of a device without a driver.
+ * @dev: Device without a drvier.
+ *
+ * Delete all non-persistent links from this device to any suppliers.
+ *
+ * Persistent links stay around, but their status is changed to "available",
+ * unless they already are in the "supplier unbind in progress" state in which
+ * case they need not be updated.
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ */
+void device_links_no_driver(struct device *dev)
+{
+	struct device_link *link, *ln;
+
+	mutex_lock(&device_links_lock);
+
+	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_suppliers, c_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		if (link->flags & DEVICE_LINK_AUTOREMOVE) {
+			__device_link_del(link);
+		} else {
+			spin_lock(&link->lock);
+
+			if (link->status != DEVICE_LINK_SUPPLIER_UNBIND)
+				link->status = DEVICE_LINK_AVAILABLE;
+
+			spin_unlock(&link->lock);
+		}
+	}
+
+	mutex_unlock(&device_links_lock);
+}
+
+/**
+ * device_links_busy - Check if there are any busy links to consumers.
+ * @dev: Device to check.
+ *
+ * Check each consumer of the device and return 'true' if its link's status
+ * is one of "consumer probe" or "active" (meaning that the given consumer is
+ * probing right now or its driver is present).  Otherwise, change the link
+ * state to "supplier unbind" to prevent the consumer from being probed
+ * successfully going forward.
+ *
+ * Return 'false' if there are no probing or active consumers.
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ */
+bool device_links_busy(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+	bool ret = false;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		spin_lock(&link->lock);
+		if (link->status == DEVICE_LINK_CONSUMER_PROBE
+		    || link->status == DEVICE_LINK_ACTIVE) {
+			spin_unlock(&link->lock);
+			ret = true;
+			break;
+		}
+		link->status = DEVICE_LINK_SUPPLIER_UNBIND;
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+	return ret;
+}
+
+/**
+ * device_links_unbind_consumers - Force unbind consumers of the given device.
+ * @dev: Device to unbind the consumers of.
+ *
+ * Walk the list of links to consumers for @dev and if any of them is in the
+ * "consumer probe" state, wait for all device probes in progress to complete
+ * and start over.
+ *
+ * If that's not the case, change the status of the link to "supplier unbind"
+ * and check if the link was in the "active" state.  If so, force the consumer
+ * driver to unbind and start over (the consumer will not re-probe as we have
+ * changed the state of the link already).
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ */
+void device_links_unbind_consumers(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+ start:
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
+		enum device_link_status status;
+
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		spin_lock(&link->lock);
+		status = link->status;
+		if (status == DEVICE_LINK_CONSUMER_PROBE) {
+			spin_unlock(&link->lock);
+
+			device_links_read_unlock(idx);
+
+			wait_for_device_probe();
+			goto start;
+		}
+		link->status = DEVICE_LINK_SUPPLIER_UNBIND;
+		if (status == DEVICE_LINK_ACTIVE) {
+			struct device *consumer = link->consumer;
+
+			get_device(consumer);
+			spin_unlock(&link->lock);
+
+			device_links_read_unlock(idx);
+
+			device_release_driver_internal(consumer, NULL,
+						       consumer->parent);
+			put_device(consumer);
+			goto start;
+		}
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+}
+
+/* Device links support end. */
+
 int (*platform_notify)(struct device *dev) = NULL;
 int (*platform_notify_remove)(struct device *dev) = NULL;
 static struct kobject *dev_kobj;
@@ -711,6 +1156,8 @@ void device_initialize(struct device *de
 #ifdef CONFIG_GENERIC_MSI_IRQ
 	INIT_LIST_HEAD(&dev->msi_list);
 #endif
+	INIT_LIST_HEAD(&dev->links_to_consumers);
+	INIT_LIST_HEAD(&dev->links_to_suppliers);
 }
 EXPORT_SYMBOL_GPL(device_initialize);
 
@@ -1233,6 +1680,7 @@ void device_del(struct device *dev)
 {
 	struct device *parent = dev->parent;
 	struct class_interface *class_intf;
+	struct device_link *link, *ln;
 
 	/* Notify clients of device removal.  This call must come
 	 * before dpm_sysfs_remove().
@@ -1240,6 +1688,30 @@ void device_del(struct device *dev)
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DEL_DEVICE, dev);
+
+	/*
+	 * Delete all of the remaining links from this device to any other
+	 * devices (either consumers or suppliers).
+	 *
+	 * This requires that all links be dormant, so warn if that's no the
+	 * case.
+	 */
+	mutex_lock(&device_links_lock);
+
+	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_suppliers, c_node) {
+		WARN_ON(link->status != DEVICE_LINK_DORMANT &&
+			!(link->flags & DEVICE_LINK_STATELESS));
+		__device_link_del(link);
+	}
+
+	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_consumers, s_node) {
+		WARN_ON(link->status != DEVICE_LINK_DORMANT &&
+			!(link->flags & DEVICE_LINK_STATELESS));
+		__device_link_del(link);
+	}
+
+	mutex_unlock(&device_links_lock);
+
 	dpm_sysfs_remove(dev);
 	if (parent)
 		klist_del(&dev->p->knode_parent);
Index: linux-pm/drivers/base/dd.c
===================================================================
--- linux-pm.orig/drivers/base/dd.c
+++ linux-pm/drivers/base/dd.c
@@ -249,6 +249,7 @@ static void driver_bound(struct device *
 		 __func__, dev_name(dev));
 
 	klist_add_tail(&dev->p->knode_driver, &dev->driver->p->klist_devices);
+	device_links_driver_bound(dev);
 
 	device_pm_check_callbacks(dev);
 
@@ -399,6 +400,7 @@ probe_failed:
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
 pinctrl_bind_failed:
+	device_links_no_driver(dev);
 	devres_release_all(dev);
 	driver_sysfs_remove(dev);
 	dev->driver = NULL;
@@ -489,6 +491,10 @@ int driver_probe_device(struct device_dr
 	if (!device_is_registered(dev))
 		return -ENODEV;
 
+	ret = device_links_check_suppliers(dev);
+	if (ret)
+		return ret;
+
 	pr_debug("bus: '%s': %s: matched device %s with driver %s\n",
 		 drv->bus->name, __func__, dev_name(dev), drv->name);
 
@@ -756,7 +762,7 @@ EXPORT_SYMBOL_GPL(driver_attach);
  * __device_release_driver() must be called with @dev lock held.
  * When called for a USB interface, @dev->parent lock must be held as well.
  */
-static void __device_release_driver(struct device *dev)
+static void __device_release_driver(struct device *dev, struct device *parent)
 {
 	struct device_driver *drv;
 
@@ -765,6 +771,25 @@ static void __device_release_driver(stru
 		if (driver_allows_async_probing(drv))
 			async_synchronize_full();
 
+		while (device_links_busy(dev)) {
+			device_unlock(dev);
+			if (parent)
+				device_unlock(parent);
+
+			device_links_unbind_consumers(dev);
+			if (parent)
+				device_lock(parent);
+
+			device_lock(dev);
+			/*
+			 * A concurrent invocation of the same function might
+			 * have released the driver successfully while this one
+			 * was waiting, so check for that.
+			 */
+			if (dev->driver != drv)
+				return;
+		}
+
 		pm_runtime_get_sync(dev);
 
 		driver_sysfs_remove(dev);
@@ -780,6 +805,9 @@ static void __device_release_driver(stru
 			dev->bus->remove(dev);
 		else if (drv->remove)
 			drv->remove(dev);
+
+		device_links_driver_gone(dev);
+		device_links_no_driver(dev);
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);
@@ -796,16 +824,16 @@ static void __device_release_driver(stru
 	}
 }
 
-static void device_release_driver_internal(struct device *dev,
-					   struct device_driver *drv,
-					   struct device *parent)
+void device_release_driver_internal(struct device *dev,
+				    struct device_driver *drv,
+				    struct device *parent)
 {
 	if (parent)
 		device_lock(parent);
 
 	device_lock(dev);
 	if (!drv || drv == dev->driver)
-		__device_release_driver(dev);
+		__device_release_driver(dev, parent);
 
 	device_unlock(dev);
 	if (parent)
@@ -818,6 +846,10 @@ static void device_release_driver_intern
  *
  * Manually detach device from driver.
  * When called for a USB interface, @dev->parent lock must be held.
+ *
+ * If this function is to be called with @dev->parent lock held, ensure that
+ * the device's consumers are unbound in advance or that their locks can be
+ * acquired under the @dev->parent lock.
  */
 void device_release_driver(struct device *dev)
 {
Index: linux-pm/include/linux/device.h
===================================================================
--- linux-pm.orig/include/linux/device.h
+++ linux-pm/include/linux/device.h
@@ -706,6 +706,35 @@ struct device_dma_parameters {
 	unsigned long segment_boundary_mask;
 };
 
+enum device_link_status {
+	DEVICE_LINK_NO_STATE = -1,
+	DEVICE_LINK_DORMANT = 0,	/* Link not in use. */
+	DEVICE_LINK_AVAILABLE,		/* Supplier driver is present. */
+	DEVICE_LINK_ACTIVE,		/* Consumer driver is present too. */
+	DEVICE_LINK_CONSUMER_PROBE,	/* Consumer is probing. */
+	DEVICE_LINK_SUPPLIER_UNBIND,	/* Supplier is unbinding. */
+};
+
+/*
+ * Device link flags.
+ *
+ * STATELESS: The state machine is not applicable to this link.
+ * AUTOREMOVE: Remove this link automatically on cunsumer driver unbind.
+ */
+#define DEVICE_LINK_STATELESS	(1 << 0)
+#define DEVICE_LINK_AUTOREMOVE	(1 << 1)
+
+struct device_link {
+	struct device *supplier;
+	struct list_head s_node;
+	struct device *consumer;
+	struct list_head c_node;
+	enum device_link_status status;
+	u32 flags;
+	spinlock_t lock;
+	struct rcu_head rcu_head;
+};
+
 /**
  * struct device - The basic device structure
  * @parent:	The device's "parent" device, the device to which it is attached.
@@ -731,6 +760,8 @@ struct device_dma_parameters {
  * 		on.  This shrinks the "Board Support Packages" (BSPs) and
  * 		minimizes board-specific #ifdefs in drivers.
  * @driver_data: Private pointer for driver specific info.
+ * @links_to_consumers: Links to consumer devices.
+ * @links_to_suppliers: Links to supplier devices.
  * @power:	For device power management.
  * 		See Documentation/power/devices.txt for details.
  * @pm_domain:	Provide callbacks that are executed during system suspend,
@@ -797,6 +828,8 @@ struct device {
 					   core doesn't touch it */
 	void		*driver_data;	/* Driver data, set and get with
 					   dev_set/get_drvdata */
+	struct list_head	links_to_consumers;
+	struct list_head	links_to_suppliers;
 	struct dev_pm_info	power;
 	struct dev_pm_domain	*pm_domain;
 
@@ -1113,6 +1146,11 @@ extern void device_shutdown(void);
 /* debugging and troubleshooting/diagnostic helpers. */
 extern const char *dev_driver_string(const struct device *dev);
 
+/* Device links interface. */
+struct device_link *device_link_add(struct device *consumer,
+				    struct device *supplier,
+				    enum device_link_status status, u32 flags);
+void device_link_del(struct device_link *link);
 
 #ifdef CONFIG_PRINTK
 

Re: [Update][RFC/RFT][PATCH v3 2/5] driver core: Functional dependencies tracking support

[Lukas Wunner]

On Fri, Sep 16, 2016 at 02:33:55PM +0200, Rafael J. Wysocki wrote:
> From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> 
> Currently, there is a problem with taking functional dependencies
> between into account.
         ^
	 devices

> What I mean by a "functional dependency" is when the driver of device
> B needs device A to be functional and (generally) its driver to be
> present in order to work properly.  This has certain consequences
> for power management (suspend/resume and runtime PM ordering) and
> shutdown ordering of these devices.  In general, it also implies that
> the driver of A needs to be working for B to be probed successfully
> and it cannot be unbound from the device before the B's driver.  
                                                                 ^^
                                                Trailing whitespace.

> Support for representing those functional dependencies between
> devices is added here to allow the driver core to track them and act
> on them in certain cases where applicable.
> 
> The argument for doing that in the driver core is that there are
> quite a few distinct use cases involving device dependencies, they
> are relatively hard to get right in a driver (if one wants to
> address all of them properly) and it only gets worse if multiplied
> by the number of drivers potentially needing to do it.  Morever, at
> least one case (asynchronous system suspend/resume) cannot be handled
> in a single driver at all, because it requires the driver of A to
> wait for B to suspend (during system suspend) and the driver of B to
> wait for A to resume (during system resume).
> 
> For this reason, represent dependencies between devices as "links",
> with the help of struct device_link objects each containing pointers
> to the "linked" devices, a list node for each of them, status
> information, flags, a lock and an RCU head for synchronization.
> 
> Also add two new list heads, links_to_consumers and links_to_suppliers,
> to struct device to represent the lists of links to the devices that
> depend on the given one (consumers) and to the devices depended on
> by it (suppliers), respectively.
> 
> The entire data structure consisting of all of the lists of link
> objects for all devices is protected by SRCU (for list walking)
> and a by mutex (for link object addition/removal).  In addition
      ^^^^
      by a

> to that, each link object has an internal status field whose
> value reflects what's happening to the devices pointed to by
> the link.  That status field is protected by an internal spinlock.

More precisely, the status field tracks the driver boundness of
the two devices comprising the link.  ("what's happening to the
devices" is a bit broad, this is really about the *drivers*.)

> New links are added by calling device_link_add() which takes four
> arguments: pointers to the devices in question, the initial status
> of the link and flags.  In particular, if DEVICE_LINK_STATELESS is
> set in the flags, the link status is not to be taken into account
> for this link and the driver core will not manage it.  In turn, if
> DEVICE_LINK_AUTOREMOVE is set in the flags, the driver core will
> remove the link automatically when the consumer device driver
> unbinds from it.
> 
> One of the actions carried out by device_link_add() is to reorder
> the lists used for device shutdown and system suspend/resume to
> put the consumer device along with all of its children and all of
> its consumers (and so on, recursively) to the ends of those list
                                                                  ^
                                                                  s

> in order to ensure the right ordering between all of the supplier
> and consumer devices.
> 
> For this reason, it is not possible to create a link between two
> devices if the would-be supplier device already depends on the
> would-be consumer device as either a direct descendant of it or a
> consumer of one of its direct descendants or one of its consumers
> and so on.
> 
> It also is impossible to create a link between a parent and a child
> device (in any direction).
>
> There are two types of link objects, persistent and non-persistent.
> The persistent ones stay around until one of the target devices is
> deleted, while the non-persistent ones are removed automatically when
> the consumer driver unbinds from its device (ie. they are assumed to
> be valid only as long as the consumer device has a driver bound to
> it).  Persistent links are created by default and non-persistent
> links are created when the DEVICE_LINK_AUTOREMOVE flag is passed
> to device_link_add().
> 
> Both persistent and non-persistent device links can be deleted
> explicitly with the help of device_link_del().
> 
> Links created without the DEVICE_LINK_STATELESS flag set are managed
> by the driver core using a simple state machine.  There are 5 states
> each link can be in: DORMANT (unused), AVAILABLE (the supplier driver
> is present and functional), CONSUMER_PROBE (the consumer driver is
> probing), ACTIVE (both supplier and consumer drivers are present and
> functional), and SUPPLIER_UNBIND (the supplier driver is unbinding).
> The driver core updates the link state automatically depending on
> what happens to the linked devices and for each link state specific
> actions are taken in addition to that.
> 
> For example, if the supplier driver unbinds from its device, the
> driver core will also unbind the drivers of all of its consumers
> automatically under the assumption that they cannot function
> properly without the supplier.  Analogously, the driver core will
> only allow the consumer driver to bind to its device is the
                                                       ^^
                                                       if

> supplier driver is present and functional (ie. the link is in
> the AVAILABLE state).  If that's not the case, it will rely on
> the existing deferred probing mechanism to wait for the supplier
> driver to become available.
> 
> Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> ---
[snip]
> +/**
> + * device_link_add - Create a link between two devices.
> + * @consumer: Consumer end of the link.
> + * @supplier: Supplier end of the link.
> + * @status: The initial status of the link.
> + * @flags: Link flags.
> + *
> + * If the DEVICE_LINK_STATELESS flag is set, @status is ignored.  Otherwise,
> + * the caller is responsible for ensuring that @status reflects the current
> + * status of both @consumer and @supplier.
> + *
> + * If the DEVICE_LINK_AUTOREMOVE is set, the link will be removed automatically
> + * when the consumer device driver unbinds from it.  The combination of both
> + * DEVICE_LINK_AUTOREMOVE and DEVICE_LINK_STATELESS set is invalid and will
> + * cause NULL to be returned.
> + *
> + * A side effect of the link creation is re-ordering of dpm_list and the
> + * devices_kset list by moving the consumer device and all devices depending
> + * on it to the ends of these lists.
> + */
> +struct device_link *device_link_add(struct device *consumer,
> +				    struct device *supplier,
> +				    enum device_link_status status, u32 flags)
> +{
> +	struct device_link *link;
> +
> +	if (!consumer || !supplier || supplier == consumer->parent ||

So a link from the child to the parent is forbidden, but e.g. to the
grandparent is not.  Either this should be forbidden for all ancestors
of the child or none.  I'd vote for the latter.  I don't see a reason
why this shouldn't be allowed.  It doesn't introduce a dependency loop
and it might be useful if someone needs a driver presence dependency
between parent and child.

[snip]
> +static int device_links_read_lock(void)
> +{
> +	return srcu_read_lock(&device_links_srcu);
> +}
> +
> +static void device_links_read_unlock(int idx)
> +{
> +	return srcu_read_unlock(&device_links_srcu, idx);
> +}

How about declaring the above two functions inline?

[snip]
> +/**
> + * device_links_check_suppliers - Check supplier devices for this one.

The short description is mostly a repetition of the function name
and thus not very informative.

Imagine someone coming here from driver_probe_device(), where this
function is invoked.  They should immediately get an idea what
the function does.

How about: "Check presence of supplier drivers"

> + * @dev: Consumer device.
> + *
> + * Check links from this device to any suppliers.  Walk the list of the device's
> + * consumer links and see if all of the suppliers are available.  If not, simply
      ^^^^^^^^
"supplier links and see if all if them are available."

[snip]
> +/**
> + * device_links_no_driver - Update links of a device without a driver.
> + * @dev: Device without a drvier.
> + *
> + * Delete all non-persistent links from this device to any suppliers.
> + *
> + * Persistent links stay around, but their status is changed to "available",
> + * unless they already are in the "supplier unbind in progress" state in which
> + * case they need not be updated.

Kerneldoc refers to persistent links, a term which no longer exists in v3.

[snip]
> +void device_links_no_driver(struct device *dev)
> +{
> +	struct device_link *link, *ln;
> +
> +	mutex_lock(&device_links_lock);
> +
> +	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_suppliers, c_node) {
> +		if (link->flags & DEVICE_LINK_STATELESS)
> +			continue;
> +
> +		if (link->flags & DEVICE_LINK_AUTOREMOVE) {
> +			__device_link_del(link);

The link will be autoremoved not only when the consumer unbinds,
but also when probing the consumer fails.

Looks like a bug.

Without the autoremove functionality, this function shrinks to just a
few LoC and it's probably okay to just duplicate that code and copy it
into device_links_driver_gone().  Then you'd have two separate
functions, one for the error path in really_probe() and the other for
__device_release_driver().

And you could then also give the functions names that match where
they're called from, e.g. device_links_probe_failed() and
device_links_driver_unbound().  Because the existing names
device_links_driver_gone() and device_links_no_driver() are very
similar and thus a bit confusing.

[snip]
> +void device_links_unbind_consumers(struct device *dev)
> +{
> +	struct device_link *link;
> +	int idx;
> +
> + start:
> +	idx = device_links_read_lock();
> +
> +	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
> +		enum device_link_status status;
> +
> +		if (link->flags & DEVICE_LINK_STATELESS)
> +			continue;
> +
> +		spin_lock(&link->lock);
> +		status = link->status;
> +		if (status == DEVICE_LINK_CONSUMER_PROBE) {
> +			spin_unlock(&link->lock);
> +
> +			device_links_read_unlock(idx);
> +
> +			wait_for_device_probe();
> +			goto start;
> +		}
> +		link->status = DEVICE_LINK_SUPPLIER_UNBIND;
> +		if (status == DEVICE_LINK_ACTIVE) {
> +			struct device *consumer = link->consumer;
> +
> +			get_device(consumer);

As long as the struct device_link exists, a ref is held on the
supplier and consumer.  Why acquire another ref here?

> +			spin_unlock(&link->lock);

The lock is released both at the beginning of this if-block and
immediately after the if-block (in case the if-condition is false).
Why not simply release the lock *before* the if-block?

[snip]
> @@ -1233,6 +1680,7 @@ void device_del(struct device *dev)
>  {
>  	struct device *parent = dev->parent;
>  	struct class_interface *class_intf;
> +	struct device_link *link, *ln;
>  
>  	/* Notify clients of device removal.  This call must come
>  	 * before dpm_sysfs_remove().
> @@ -1240,6 +1688,30 @@ void device_del(struct device *dev)
>  	if (dev->bus)
>  		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
>  					     BUS_NOTIFY_DEL_DEVICE, dev);
> +
> +	/*
> +	 * Delete all of the remaining links from this device to any other
> +	 * devices (either consumers or suppliers).
> +	 *
> +	 * This requires that all links be dormant, so warn if that's no the
> +	 * case.
> +	 */

How about moving this to a separate function, e.g. device_links_purge(dev)?

In all other cases you've created a new function even though it's only
called from a single place, which makes sense because it avoids cluttering
up these central functions like device_del().

> +	mutex_lock(&device_links_lock);
> +
> +	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_suppliers, c_node) {
> +		WARN_ON(link->status != DEVICE_LINK_DORMANT &&
> +			!(link->flags & DEVICE_LINK_STATELESS));
> +		__device_link_del(link);
> +	}

Shouldn't it also be legal for the supplier links to be in
DEVICE_LINK_AVAILABLE state upon removal of a consumer device?

(And perhaps also DEVICE_LINK_SUPPLIER_UNBIND?)

Looks like a bug.

[snip]
> --- linux-pm.orig/drivers/base/dd.c
> +++ linux-pm/drivers/base/dd.c
> @@ -249,6 +249,7 @@ static void driver_bound(struct device *
>  		 __func__, dev_name(dev));
>  
>  	klist_add_tail(&dev->p->knode_driver, &dev->driver->p->klist_devices);
> +	device_links_driver_bound(dev);
>  
>  	device_pm_check_callbacks(dev);
>  
> @@ -399,6 +400,7 @@ probe_failed:
>  		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
>  					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
>  pinctrl_bind_failed:
> +	device_links_no_driver(dev);
>  	devres_release_all(dev);
>  	driver_sysfs_remove(dev);
>  	dev->driver = NULL;
> @@ -489,6 +491,10 @@ int driver_probe_device(struct device_dr
>  	if (!device_is_registered(dev))
>  		return -ENODEV;
>  
> +	ret = device_links_check_suppliers(dev);
> +	if (ret)
> +		return ret;
> +

I think clarity would improve if you would move the call to
device_links_check_suppliers() down the call stack into really_probe().
Then it would be in the same place as the call to device_links_no_driver()
(if probing fails).

Furthermore, driver_probe_device() runtime resumes the parent before
probing a device, but you're not doing the same for the suppliers.
Looks like a bug.

[snip]
> --- linux-pm.orig/include/linux/device.h
> +++ linux-pm/include/linux/device.h
> @@ -706,6 +706,35 @@ struct device_dma_parameters {
>  	unsigned long segment_boundary_mask;
>  };
>  
> +enum device_link_status {
> +	DEVICE_LINK_NO_STATE = -1,

How about a comment like /* Stateless. */ so that the relationship to
the DEVICE_LINK_STATELESS flag is clear.

> +	DEVICE_LINK_DORMANT = 0,	/* Link not in use. */
> +	DEVICE_LINK_AVAILABLE,		/* Supplier driver is present. */
> +	DEVICE_LINK_ACTIVE,		/* Consumer driver is present too. */
> +	DEVICE_LINK_CONSUMER_PROBE,	/* Consumer is probing. */
> +	DEVICE_LINK_SUPPLIER_UNBIND,	/* Supplier is unbinding. */
> +};
> +
> +/*
> + * Device link flags.
> + *
> + * STATELESS: The state machine is not applicable to this link.

IMO the consequence of setting this flag is not immediately clear from the
comment. How about: "Ignore driver presence."

> + * AUTOREMOVE: Remove this link automatically on cunsumer driver unbind.
                                                     ^
                                                     o

Thanks,

Lukas

Re: [Update][RFC/RFT][PATCH v3 2/5] driver core: Functional dependencies tracking support

[Lukas Wunner]

On Tue, Sep 20, 2016 at 12:46:30AM +0200, Lukas Wunner wrote:
> On Fri, Sep 16, 2016 at 02:33:55PM +0200, Rafael J. Wysocki wrote:
> > +void device_links_unbind_consumers(struct device *dev)
> > +{
> > +	struct device_link *link;
> > +	int idx;
> > +
> > + start:
> > +	idx = device_links_read_lock();
> > +
> > +	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
> > +		enum device_link_status status;
> > +
> > +		if (link->flags & DEVICE_LINK_STATELESS)
> > +			continue;
> > +
> > +		spin_lock(&link->lock);
> > +		status = link->status;
> > +		if (status == DEVICE_LINK_CONSUMER_PROBE) {
> > +			spin_unlock(&link->lock);
> > +
> > +			device_links_read_unlock(idx);
> > +
> > +			wait_for_device_probe();
> > +			goto start;
> > +		}
> > +		link->status = DEVICE_LINK_SUPPLIER_UNBIND;
> > +		if (status == DEVICE_LINK_ACTIVE) {
> > +			struct device *consumer = link->consumer;
> > +
> > +			get_device(consumer);
> 
> As long as the struct device_link exists, a ref is held on the
> supplier and consumer.  Why acquire another ref here?

I'm withdrawing this particular comment as I failed to see that
device_links_read_unlock() is called next, so nothing prevents
the device link from being deleted, same for the consumer, thus
the ref needs to be acquired for device_release_driver_internal()
and this portion of Rafael's code seems perfectly correct.

Thanks & sorry for the noise,

Lukas

Re: [Update][RFC/RFT][PATCH v3 2/5] driver core: Functional dependencies tracking support

[Rafael J. Wysocki]

On Tuesday, September 20, 2016 12:46:30 AM Lukas Wunner wrote:
> On Fri, Sep 16, 2016 at 02:33:55PM +0200, Rafael J. Wysocki wrote:
> > From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> > 
> > Currently, there is a problem with taking functional dependencies
> > between into account.
>          ^
> 	 devices
> 
> > What I mean by a "functional dependency" is when the driver of device
> > B needs device A to be functional and (generally) its driver to be
> > present in order to work properly.  This has certain consequences
> > for power management (suspend/resume and runtime PM ordering) and
> > shutdown ordering of these devices.  In general, it also implies that
> > the driver of A needs to be working for B to be probed successfully
> > and it cannot be unbound from the device before the B's driver.  
>                                                                  ^^
>                                                 Trailing whitespace.
> 
> > Support for representing those functional dependencies between
> > devices is added here to allow the driver core to track them and act
> > on them in certain cases where applicable.
> > 
> > The argument for doing that in the driver core is that there are
> > quite a few distinct use cases involving device dependencies, they
> > are relatively hard to get right in a driver (if one wants to
> > address all of them properly) and it only gets worse if multiplied
> > by the number of drivers potentially needing to do it.  Morever, at
> > least one case (asynchronous system suspend/resume) cannot be handled
> > in a single driver at all, because it requires the driver of A to
> > wait for B to suspend (during system suspend) and the driver of B to
> > wait for A to resume (during system resume).
> > 
> > For this reason, represent dependencies between devices as "links",
> > with the help of struct device_link objects each containing pointers
> > to the "linked" devices, a list node for each of them, status
> > information, flags, a lock and an RCU head for synchronization.
> > 
> > Also add two new list heads, links_to_consumers and links_to_suppliers,
> > to struct device to represent the lists of links to the devices that
> > depend on the given one (consumers) and to the devices depended on
> > by it (suppliers), respectively.
> > 
> > The entire data structure consisting of all of the lists of link
> > objects for all devices is protected by SRCU (for list walking)
> > and a by mutex (for link object addition/removal).  In addition
>       ^^^^
>       by a
> 
> > to that, each link object has an internal status field whose
> > value reflects what's happening to the devices pointed to by
> > the link.  That status field is protected by an internal spinlock.
> 
> More precisely, the status field tracks the driver boundness of
> the two devices comprising the link.  ("what's happening to the
> devices" is a bit broad, this is really about the *drivers*.)

Basically OK, but it will be more than that most likely due to the bug
reported by Marek.

And thanks for the typo fixes. :-)

> > New links are added by calling device_link_add() which takes four
> > arguments: pointers to the devices in question, the initial status
> > of the link and flags.  In particular, if DEVICE_LINK_STATELESS is
> > set in the flags, the link status is not to be taken into account
> > for this link and the driver core will not manage it.  In turn, if
> > DEVICE_LINK_AUTOREMOVE is set in the flags, the driver core will
> > remove the link automatically when the consumer device driver
> > unbinds from it.
> > 
> > One of the actions carried out by device_link_add() is to reorder
> > the lists used for device shutdown and system suspend/resume to
> > put the consumer device along with all of its children and all of
> > its consumers (and so on, recursively) to the ends of those list
>                                                                   ^
>                                                                   s
> 
> > in order to ensure the right ordering between all of the supplier
> > and consumer devices.
> > 
> > For this reason, it is not possible to create a link between two
> > devices if the would-be supplier device already depends on the
> > would-be consumer device as either a direct descendant of it or a
> > consumer of one of its direct descendants or one of its consumers
> > and so on.
> > 
> > It also is impossible to create a link between a parent and a child
> > device (in any direction).
> >
> > There are two types of link objects, persistent and non-persistent.
> > The persistent ones stay around until one of the target devices is
> > deleted, while the non-persistent ones are removed automatically when
> > the consumer driver unbinds from its device (ie. they are assumed to
> > be valid only as long as the consumer device has a driver bound to
> > it).  Persistent links are created by default and non-persistent
> > links are created when the DEVICE_LINK_AUTOREMOVE flag is passed
> > to device_link_add().
> > 
> > Both persistent and non-persistent device links can be deleted
> > explicitly with the help of device_link_del().
> > 
> > Links created without the DEVICE_LINK_STATELESS flag set are managed
> > by the driver core using a simple state machine.  There are 5 states
> > each link can be in: DORMANT (unused), AVAILABLE (the supplier driver
> > is present and functional), CONSUMER_PROBE (the consumer driver is
> > probing), ACTIVE (both supplier and consumer drivers are present and
> > functional), and SUPPLIER_UNBIND (the supplier driver is unbinding).
> > The driver core updates the link state automatically depending on
> > what happens to the linked devices and for each link state specific
> > actions are taken in addition to that.
> > 
> > For example, if the supplier driver unbinds from its device, the
> > driver core will also unbind the drivers of all of its consumers
> > automatically under the assumption that they cannot function
> > properly without the supplier.  Analogously, the driver core will
> > only allow the consumer driver to bind to its device is the
>                                                        ^^
>                                                        if
> 
> > supplier driver is present and functional (ie. the link is in
> > the AVAILABLE state).  If that's not the case, it will rely on
> > the existing deferred probing mechanism to wait for the supplier
> > driver to become available.
> > 
> > Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> > Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> > ---
> [snip]
> > +/**
> > + * device_link_add - Create a link between two devices.
> > + * @consumer: Consumer end of the link.
> > + * @supplier: Supplier end of the link.
> > + * @status: The initial status of the link.
> > + * @flags: Link flags.
> > + *
> > + * If the DEVICE_LINK_STATELESS flag is set, @status is ignored.  Otherwise,
> > + * the caller is responsible for ensuring that @status reflects the current
> > + * status of both @consumer and @supplier.
> > + *
> > + * If the DEVICE_LINK_AUTOREMOVE is set, the link will be removed automatically
> > + * when the consumer device driver unbinds from it.  The combination of both
> > + * DEVICE_LINK_AUTOREMOVE and DEVICE_LINK_STATELESS set is invalid and will
> > + * cause NULL to be returned.
> > + *
> > + * A side effect of the link creation is re-ordering of dpm_list and the
> > + * devices_kset list by moving the consumer device and all devices depending
> > + * on it to the ends of these lists.
> > + */
> > +struct device_link *device_link_add(struct device *consumer,
> > +				    struct device *supplier,
> > +				    enum device_link_status status, u32 flags)
> > +{
> > +	struct device_link *link;
> > +
> > +	if (!consumer || !supplier || supplier == consumer->parent ||
> 
> So a link from the child to the parent is forbidden, but e.g. to the
> grandparent is not.  Either this should be forbidden for all ancestors
> of the child or none.  I'd vote for the latter.  I don't see a reason
> why this shouldn't be allowed.  It doesn't introduce a dependency loop
> and it might be useful if someone needs a driver presence dependency
> between parent and child.

While I can remove this check, I don't see why anyone would use links for
that instead of adding a device flag to cause the core to wait for the
parent to be probed if the child requires that.

> [snip]
> > +static int device_links_read_lock(void)
> > +{
> > +	return srcu_read_lock(&device_links_srcu);
> > +}
> > +
> > +static void device_links_read_unlock(int idx)
> > +{
> > +	return srcu_read_unlock(&device_links_srcu, idx);
> > +}
> 
> How about declaring the above two functions inline?

Well, does it really matter?  The complier should be able to make them inline
anyway if it sees the point.

> [snip]
> > +/**
> > + * device_links_check_suppliers - Check supplier devices for this one.
> 
> The short description is mostly a repetition of the function name
> and thus not very informative.
> 
> Imagine someone coming here from driver_probe_device(), where this
> function is invoked.  They should immediately get an idea what
> the function does.
> 
> How about: "Check presence of supplier drivers"

Sounds good.

> > + * @dev: Consumer device.
> > + *
> > + * Check links from this device to any suppliers.  Walk the list of the device's
> > + * consumer links and see if all of the suppliers are available.  If not, simply
>       ^^^^^^^^
> "supplier links and see if all if them are available."
> 
> [snip]
> > +/**
> > + * device_links_no_driver - Update links of a device without a driver.
> > + * @dev: Device without a drvier.
> > + *
> > + * Delete all non-persistent links from this device to any suppliers.
> > + *
> > + * Persistent links stay around, but their status is changed to "available",
> > + * unless they already are in the "supplier unbind in progress" state in which
> > + * case they need not be updated.
> 
> Kerneldoc refers to persistent links, a term which no longer exists in v3.

It does exist.

The flag is not present any more, but the links may still be persistent
(and the changelog talks about "persistent" and "non-persistent" links too).

> [snip]
> > +void device_links_no_driver(struct device *dev)
> > +{
> > +	struct device_link *link, *ln;
> > +
> > +	mutex_lock(&device_links_lock);
> > +
> > +	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_suppliers, c_node) {
> > +		if (link->flags & DEVICE_LINK_STATELESS)
> > +			continue;
> > +
> > +		if (link->flags & DEVICE_LINK_AUTOREMOVE) {
> > +			__device_link_del(link);
> 
> The link will be autoremoved not only when the consumer unbinds,
> but also when probing the consumer fails.
> 
> Looks like a bug.

It really was intentional, because the use case I see for AUTOREMOVE (and
the only one to be honest) is when the link is created by the consumer
probe in which case it wants to avoid worrying about the cleanup part.

Which also is applicable to the cleanup when the probe fails IMO.

> Without the autoremove functionality, this function shrinks to just a
> few LoC and it's probably okay to just duplicate that code and copy it
> into device_links_driver_gone().  Then you'd have two separate
> functions, one for the error path in really_probe() and the other for
> __device_release_driver().
> 
> And you could then also give the functions names that match where
> they're called from, e.g. device_links_probe_failed() and
> device_links_driver_unbound().  Because the existing names
> device_links_driver_gone() and device_links_no_driver() are very
> similar and thus a bit confusing.

I can drop AUTOREMOVE entirely if it is problematic.

> [snip]
> > +void device_links_unbind_consumers(struct device *dev)
> > +{
> > +	struct device_link *link;
> > +	int idx;
> > +
> > + start:
> > +	idx = device_links_read_lock();
> > +
> > +	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
> > +		enum device_link_status status;
> > +
> > +		if (link->flags & DEVICE_LINK_STATELESS)
> > +			continue;
> > +
> > +		spin_lock(&link->lock);
> > +		status = link->status;
> > +		if (status == DEVICE_LINK_CONSUMER_PROBE) {
> > +			spin_unlock(&link->lock);
> > +
> > +			device_links_read_unlock(idx);
> > +
> > +			wait_for_device_probe();
> > +			goto start;
> > +		}
> > +		link->status = DEVICE_LINK_SUPPLIER_UNBIND;
> > +		if (status == DEVICE_LINK_ACTIVE) {
> > +			struct device *consumer = link->consumer;
> > +
> > +			get_device(consumer);
> 
> As long as the struct device_link exists, a ref is held on the
> supplier and consumer.  Why acquire another ref here?

This comment has just been withdrawn. :-)

> > +			spin_unlock(&link->lock);
> 
> The lock is released both at the beginning of this if-block and
> immediately after the if-block (in case the if-condition is false).
> Why not simply release the lock *before* the if-block?

Because the get_device() needs to be done under the lock.

> 
> [snip]
> > @@ -1233,6 +1680,7 @@ void device_del(struct device *dev)
> >  {
> >  	struct device *parent = dev->parent;
> >  	struct class_interface *class_intf;
> > +	struct device_link *link, *ln;
> >  
> >  	/* Notify clients of device removal.  This call must come
> >  	 * before dpm_sysfs_remove().
> > @@ -1240,6 +1688,30 @@ void device_del(struct device *dev)
> >  	if (dev->bus)
> >  		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
> >  					     BUS_NOTIFY_DEL_DEVICE, dev);
> > +
> > +	/*
> > +	 * Delete all of the remaining links from this device to any other
> > +	 * devices (either consumers or suppliers).
> > +	 *
> > +	 * This requires that all links be dormant, so warn if that's no the
> > +	 * case.
> > +	 */
> 
> How about moving this to a separate function, e.g. device_links_purge(dev)?
> 
> In all other cases you've created a new function even though it's only
> called from a single place, which makes sense because it avoids cluttering
> up these central functions like device_del().

OK

> > +	mutex_lock(&device_links_lock);
> > +
> > +	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_suppliers, c_node) {
> > +		WARN_ON(link->status != DEVICE_LINK_DORMANT &&
> > +			!(link->flags & DEVICE_LINK_STATELESS));
> > +		__device_link_del(link);
> > +	}
> 
> Shouldn't it also be legal for the supplier links to be in
> DEVICE_LINK_AVAILABLE state upon removal of a consumer device?
> 
> (And perhaps also DEVICE_LINK_SUPPLIER_UNBIND?)
> 
> Looks like a bug.

But this is done after removing the supplier driver, so the state should be
DORMANT (unless the link is stateless), shouldn't it?

> [snip]
> > --- linux-pm.orig/drivers/base/dd.c
> > +++ linux-pm/drivers/base/dd.c
> > @@ -249,6 +249,7 @@ static void driver_bound(struct device *
> >  		 __func__, dev_name(dev));
> >  
> >  	klist_add_tail(&dev->p->knode_driver, &dev->driver->p->klist_devices);
> > +	device_links_driver_bound(dev);
> >  
> >  	device_pm_check_callbacks(dev);
> >  
> > @@ -399,6 +400,7 @@ probe_failed:
> >  		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
> >  					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
> >  pinctrl_bind_failed:
> > +	device_links_no_driver(dev);
> >  	devres_release_all(dev);
> >  	driver_sysfs_remove(dev);
> >  	dev->driver = NULL;
> > @@ -489,6 +491,10 @@ int driver_probe_device(struct device_dr
> >  	if (!device_is_registered(dev))
> >  		return -ENODEV;
> >  
> > +	ret = device_links_check_suppliers(dev);
> > +	if (ret)
> > +		return ret;
> > +
> 
> I think clarity would improve if you would move the call to
> device_links_check_suppliers() down the call stack into really_probe().
> Then it would be in the same place as the call to device_links_no_driver()
> (if probing fails).

I'll have a look.

> Furthermore, driver_probe_device() runtime resumes the parent before
> probing a device, but you're not doing the same for the suppliers.
> Looks like a bug.

I need to check, but for this patch it doesn't matter anyway as links don't
apply to runtime PM at this point yet.

> [snip]
> > --- linux-pm.orig/include/linux/device.h
> > +++ linux-pm/include/linux/device.h
> > @@ -706,6 +706,35 @@ struct device_dma_parameters {
> >  	unsigned long segment_boundary_mask;
> >  };
> >  
> > +enum device_link_status {
> > +	DEVICE_LINK_NO_STATE = -1,
> 
> How about a comment like /* Stateless. */ so that the relationship to
> the DEVICE_LINK_STATELESS flag is clear.
> 
> > +	DEVICE_LINK_DORMANT = 0,	/* Link not in use. */
> > +	DEVICE_LINK_AVAILABLE,		/* Supplier driver is present. */
> > +	DEVICE_LINK_ACTIVE,		/* Consumer driver is present too. */
> > +	DEVICE_LINK_CONSUMER_PROBE,	/* Consumer is probing. */
> > +	DEVICE_LINK_SUPPLIER_UNBIND,	/* Supplier is unbinding. */
> > +};
> > +
> > +/*
> > + * Device link flags.
> > + *
> > + * STATELESS: The state machine is not applicable to this link.
> 
> IMO the consequence of setting this flag is not immediately clear from the
> comment. How about: "Ignore driver presence."

Well, I'm not sure if that's any better to be honest.  The consequences would
still be unclear with it, but for a different reason. :-)

> > + * AUTOREMOVE: Remove this link automatically on cunsumer driver unbind.

Thanks,
Rafael

Re: [Update][RFC/RFT][PATCH v3 2/5] driver core: Functional dependencies tracking support

[Lukas Wunner]

On Fri, Sep 23, 2016 at 03:42:31PM +0200, Rafael J. Wysocki wrote:
> On Tuesday, September 20, 2016 12:46:30 AM Lukas Wunner wrote:
> > On Fri, Sep 16, 2016 at 02:33:55PM +0200, Rafael J. Wysocki wrote:
> > > +void device_links_no_driver(struct device *dev)
> > > +{
> > > +	struct device_link *link, *ln;
> > > +
> > > +	mutex_lock(&device_links_lock);
> > > +
> > > +	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_suppliers, c_node) {
> > > +		if (link->flags & DEVICE_LINK_STATELESS)
> > > +			continue;
> > > +
> > > +		if (link->flags & DEVICE_LINK_AUTOREMOVE) {
> > > +			__device_link_del(link);
> > 
> > The link will be autoremoved not only when the consumer unbinds,
> > but also when probing the consumer fails.
> > 
> > Looks like a bug.
> 
> It really was intentional, because the use case I see for AUTOREMOVE (and
> the only one to be honest) is when the link is created by the consumer
> probe in which case it wants to avoid worrying about the cleanup part.
> 
> Which also is applicable to the cleanup when the probe fails IMO.

You're right, makes sense.


> > > +void device_links_unbind_consumers(struct device *dev)
> > > +{
> > > +	struct device_link *link;
> > > +	int idx;
> > > +
> > > + start:
> > > +	idx = device_links_read_lock();
> > > +
> > > +	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
> > > +		enum device_link_status status;
> > > +
> > > +		if (link->flags & DEVICE_LINK_STATELESS)
> > > +			continue;
> > > +
> > > +		spin_lock(&link->lock);
> > > +		status = link->status;
> > > +		if (status == DEVICE_LINK_CONSUMER_PROBE) {
> > > +			spin_unlock(&link->lock);
> > > +
> > > +			device_links_read_unlock(idx);
> > > +
> > > +			wait_for_device_probe();
> > > +			goto start;
> > > +		}
> > > +		link->status = DEVICE_LINK_SUPPLIER_UNBIND;
> > > +		if (status == DEVICE_LINK_ACTIVE) {
> > > +			struct device *consumer = link->consumer;
> > > +
> > > +			get_device(consumer);
> > > +			spin_unlock(&link->lock);
> > 
> > The lock is released both at the beginning of this if-block and
> > immediately after the if-block (in case the if-condition is false).
> > Why not simply release the lock *before* the if-block?
> 
> Because the get_device() needs to be done under the lock.

According to the commit message, the spinlock only protects the status
field and the consumer device is prevented from disappearing with the RCU.
So the spin lock could be released before the if-block AFAICS.
(But perhaps there are style/readability reasons to have the unlock both
in the if-block and afterwards.)


> > > @@ -1233,6 +1680,7 @@ void device_del(struct device *dev)
> > >  {
> > >  	struct device *parent = dev->parent;
> > >  	struct class_interface *class_intf;
> > > +	struct device_link *link, *ln;
> > >  
> > >  	/* Notify clients of device removal.  This call must come
> > >  	 * before dpm_sysfs_remove().
> > > @@ -1240,6 +1688,30 @@ void device_del(struct device *dev)
> > >  	if (dev->bus)
> > >  		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
> > >  					     BUS_NOTIFY_DEL_DEVICE, dev);
> > > +
> > > +	/*
> > > +	 * Delete all of the remaining links from this device to any other
> > > +	 * devices (either consumers or suppliers).
> > > +	 *
> > > +	 * This requires that all links be dormant, so warn if that's no the
> > > +	 * case.
> > > +	 */
> > > +	mutex_lock(&device_links_lock);
> > > +
> > > +	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_suppliers, c_node) {
> > > +		WARN_ON(link->status != DEVICE_LINK_DORMANT &&
> > > +			!(link->flags & DEVICE_LINK_STATELESS));
> > > +		__device_link_del(link);
> > > +	}
> > 
> > Shouldn't it also be legal for the supplier links to be in
> > DEVICE_LINK_AVAILABLE state upon removal of a consumer device?
> > 
> > (And perhaps also DEVICE_LINK_SUPPLIER_UNBIND?)
> > 
> > Looks like a bug.
> 
> But this is done after removing the supplier driver, so the state should be
> DORMANT (unless the link is stateless), shouldn't it?

The scenario I have in mind is that the supplier device is bound to a
driver and the consumer device has no driver and is being removed.
In that case the status will be DEVICE_LINK_AVAILABLE and the user
will get a WARN splat, which seems gratuitous because it should be legal.

And the other scenario is when the supplier is unbinding. It iterates
over the links to consumers and puts them in DEVICE_LINK_SUPPLIER_UNBIND.
Let's say the link to consumer A was put into that state, but there's
a consumer B remaining which is bound.  The RCU and spinlock are unlocked
before device_release_driver_internal() is called for that consumer.
If at that point consumer device A is removed for whatever reason,
the link will also be removed and the user will again get a gratuitous
WARN splat.


Thanks,

Lukas

Re: [Update][RFC/RFT][PATCH v3 2/5] driver core: Functional dependencies tracking support

[Rafael J. Wysocki]

On Mon, Sep 26, 2016 at 6:51 PM, Lukas Wunner <lukas@wunner.de> wrote:
> On Fri, Sep 23, 2016 at 03:42:31PM +0200, Rafael J. Wysocki wrote:
>> On Tuesday, September 20, 2016 12:46:30 AM Lukas Wunner wrote:
>> > On Fri, Sep 16, 2016 at 02:33:55PM +0200, Rafael J. Wysocki wrote:
>> > > +void device_links_no_driver(struct device *dev)
>> > > +{
>> > > + struct device_link *link, *ln;
>> > > +
>> > > + mutex_lock(&device_links_lock);
>> > > +
>> > > + list_for_each_entry_safe_reverse(link, ln, &dev->links_to_suppliers, c_node) {
>> > > +         if (link->flags & DEVICE_LINK_STATELESS)
>> > > +                 continue;
>> > > +
>> > > +         if (link->flags & DEVICE_LINK_AUTOREMOVE) {
>> > > +                 __device_link_del(link);
>> >
>> > The link will be autoremoved not only when the consumer unbinds,
>> > but also when probing the consumer fails.
>> >
>> > Looks like a bug.
>>
>> It really was intentional, because the use case I see for AUTOREMOVE (and
>> the only one to be honest) is when the link is created by the consumer
>> probe in which case it wants to avoid worrying about the cleanup part.
>>
>> Which also is applicable to the cleanup when the probe fails IMO.
>
> You're right, makes sense.
>
>
>> > > +void device_links_unbind_consumers(struct device *dev)
>> > > +{
>> > > + struct device_link *link;
>> > > + int idx;
>> > > +
>> > > + start:
>> > > + idx = device_links_read_lock();
>> > > +
>> > > + list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
>> > > +         enum device_link_status status;
>> > > +
>> > > +         if (link->flags & DEVICE_LINK_STATELESS)
>> > > +                 continue;
>> > > +
>> > > +         spin_lock(&link->lock);
>> > > +         status = link->status;
>> > > +         if (status == DEVICE_LINK_CONSUMER_PROBE) {
>> > > +                 spin_unlock(&link->lock);
>> > > +
>> > > +                 device_links_read_unlock(idx);
>> > > +
>> > > +                 wait_for_device_probe();
>> > > +                 goto start;
>> > > +         }
>> > > +         link->status = DEVICE_LINK_SUPPLIER_UNBIND;
>> > > +         if (status == DEVICE_LINK_ACTIVE) {
>> > > +                 struct device *consumer = link->consumer;
>> > > +
>> > > +                 get_device(consumer);
>> > > +                 spin_unlock(&link->lock);
>> >
>> > The lock is released both at the beginning of this if-block and
>> > immediately after the if-block (in case the if-condition is false).
>> > Why not simply release the lock *before* the if-block?
>>
>> Because the get_device() needs to be done under the lock.
>
> According to the commit message, the spinlock only protects the status
> field and the consumer device is prevented from disappearing with the RCU.
> So the spin lock could be released before the if-block AFAICS.
> (But perhaps there are style/readability reasons to have the unlock both
> in the if-block and afterwards.)

OK

Apparently, I was worrying about that the link might go away after the
device_links_read_unlock() in the if () block, so the object pointed
to by "consumer" had to be prevented from going away as well at that
point, but you are right that it's sufficient to call the get_device()
before the device_links_read_unlock() for that and it doesn't have to
go under the spinlock.

>> > > @@ -1233,6 +1680,7 @@ void device_del(struct device *dev)
>> > >  {
>> > >   struct device *parent = dev->parent;
>> > >   struct class_interface *class_intf;
>> > > + struct device_link *link, *ln;
>> > >
>> > >   /* Notify clients of device removal.  This call must come
>> > >    * before dpm_sysfs_remove().
>> > > @@ -1240,6 +1688,30 @@ void device_del(struct device *dev)
>> > >   if (dev->bus)
>> > >           blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
>> > >                                        BUS_NOTIFY_DEL_DEVICE, dev);
>> > > +
>> > > + /*
>> > > +  * Delete all of the remaining links from this device to any other
>> > > +  * devices (either consumers or suppliers).
>> > > +  *
>> > > +  * This requires that all links be dormant, so warn if that's no the
>> > > +  * case.
>> > > +  */
>> > > + mutex_lock(&device_links_lock);
>> > > +
>> > > + list_for_each_entry_safe_reverse(link, ln, &dev->links_to_suppliers, c_node) {
>> > > +         WARN_ON(link->status != DEVICE_LINK_DORMANT &&
>> > > +                 !(link->flags & DEVICE_LINK_STATELESS));
>> > > +         __device_link_del(link);
>> > > + }
>> >
>> > Shouldn't it also be legal for the supplier links to be in
>> > DEVICE_LINK_AVAILABLE state upon removal of a consumer device?
>> >
>> > (And perhaps also DEVICE_LINK_SUPPLIER_UNBIND?)
>> >
>> > Looks like a bug.
>>
>> But this is done after removing the supplier driver, so the state should be
>> DORMANT (unless the link is stateless), shouldn't it?
>
> The scenario I have in mind is that the supplier device is bound to a
> driver and the consumer device has no driver and is being removed.
> In that case the status will be DEVICE_LINK_AVAILABLE and the user
> will get a WARN splat, which seems gratuitous because it should be legal.
>
> And the other scenario is when the supplier is unbinding. It iterates
> over the links to consumers and puts them in DEVICE_LINK_SUPPLIER_UNBIND.
> Let's say the link to consumer A was put into that state, but there's
> a consumer B remaining which is bound.  The RCU and spinlock are unlocked
> before device_release_driver_internal() is called for that consumer.
> If at that point consumer device A is removed for whatever reason,
> the link will also be removed and the user will again get a gratuitous
> WARN splat.

Well, it all boils down to the observation that the consumer device
may be deleted when the supplier still has a driver or is unbinding,
which is a good point.

Thanks,
Rafael

Re: [Update][RFC/RFT][PATCH v3 2/5] driver core: Functional dependencies tracking support

[Lukas Wunner]

On Fri, Sep 16, 2016 at 02:33:55PM +0200, Rafael J. Wysocki wrote:
> +void device_links_unbind_consumers(struct device *dev)
> +{
> +	struct device_link *link;
> +	int idx;
> +
> + start:
> +	idx = device_links_read_lock();
> +
> +	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
> +		enum device_link_status status;
> +
> +		if (link->flags & DEVICE_LINK_STATELESS)
> +			continue;
> +
> +		spin_lock(&link->lock);
> +		status = link->status;
> +		if (status == DEVICE_LINK_CONSUMER_PROBE) {
> +			spin_unlock(&link->lock);
> +
> +			device_links_read_unlock(idx);
> +
> +			wait_for_device_probe();
> +			goto start;
> +		}

While revisiting this function it just occurred to me that there's
a theoretical infinite loop here if the consumer probes, is unbound
by the supplier, then reprobes again before the supplier had a chance
to update the link to DEVICE_LINK_SUPPLIER_UNBIND.  Perhaps this isn't
a problem in practice, but noting anyway.

The problem is that the link state is written to both by the supplier
and consumer.  If there was a separate bit in struct device_link to
indicate the supplier's desire to unbind, the problem would go away.
However a mix of such a bit plus the state machine would become
somewhat confusing...

Best regards,

Lukas

Re: [Update][RFC/RFT][PATCH v3 2/5] driver core: Functional dependencies tracking support

[Rafael J. Wysocki]

On Tue, Sep 27, 2016 at 10:54 AM, Lukas Wunner <lukas@wunner.de> wrote:
> On Fri, Sep 16, 2016 at 02:33:55PM +0200, Rafael J. Wysocki wrote:
>> +void device_links_unbind_consumers(struct device *dev)
>> +{
>> +     struct device_link *link;
>> +     int idx;
>> +
>> + start:
>> +     idx = device_links_read_lock();
>> +
>> +     list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
>> +             enum device_link_status status;
>> +
>> +             if (link->flags & DEVICE_LINK_STATELESS)
>> +                     continue;
>> +
>> +             spin_lock(&link->lock);
>> +             status = link->status;
>> +             if (status == DEVICE_LINK_CONSUMER_PROBE) {
>> +                     spin_unlock(&link->lock);
>> +
>> +                     device_links_read_unlock(idx);
>> +
>> +                     wait_for_device_probe();
>> +                     goto start;
>> +             }
>
> While revisiting this function it just occurred to me that there's
> a theoretical infinite loop here if the consumer probes, is unbound
> by the supplier, then reprobes again before the supplier had a chance
> to update the link to DEVICE_LINK_SUPPLIER_UNBIND.  Perhaps this isn't
> a problem in practice, but noting anyway.

But the consumer is unbound only after setting the link status to
DEVICE_LINK_SUPPLIER_UNBIND and then it won't probe again.

Or am I overlooking something?

Thanks,
Rafael

Re: [Update][RFC/RFT][PATCH v3 2/5] driver core: Functional dependencies tracking support

[Lukas Wunner]

On Tue, Sep 27, 2016 at 01:52:48PM +0200, Rafael J. Wysocki wrote:
> On Tue, Sep 27, 2016 at 10:54 AM, Lukas Wunner <lukas@wunner.de> wrote:
> > On Fri, Sep 16, 2016 at 02:33:55PM +0200, Rafael J. Wysocki wrote:
> >> +void device_links_unbind_consumers(struct device *dev)
> >> +{
> >> +     struct device_link *link;
> >> +     int idx;
> >> +
> >> + start:
> >> +     idx = device_links_read_lock();
> >> +
> >> +     list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
> >> +             enum device_link_status status;
> >> +
> >> +             if (link->flags & DEVICE_LINK_STATELESS)
> >> +                     continue;
> >> +
> >> +             spin_lock(&link->lock);
> >> +             status = link->status;
> >> +             if (status == DEVICE_LINK_CONSUMER_PROBE) {
> >> +                     spin_unlock(&link->lock);
> >> +
> >> +                     device_links_read_unlock(idx);
> >> +
> >> +                     wait_for_device_probe();
> >> +                     goto start;
> >> +             }
> >> +             link->status = DEVICE_LINK_SUPPLIER_UNBIND;
> >
> > While revisiting this function it just occurred to me that there's
> > a theoretical infinite loop here if the consumer probes, is unbound
> > by the supplier, then reprobes again before the supplier had a chance
> > to update the link to DEVICE_LINK_SUPPLIER_UNBIND.  Perhaps this isn't
> > a problem in practice, but noting anyway.
> 
> But the consumer is unbound only after setting the link status to
> DEVICE_LINK_SUPPLIER_UNBIND and then it won't probe again.

Sorry, looking at the code with a fresh pair of eyeballs I realize the
scenario for the infinite loop is different from what I've written above:
The infinite loop can occur if the consumer probes continuously but never
succeeds, e.g. due to some unfulfilled condition in its ->probe hook.

That could be fixed by moving the assignment

		link->status = DEVICE_LINK_SUPPLIER_UNBIND;

above the preceding if-block (but below "status = link->status;").

The next time the consumer probes, it will return with -EPROBE_DEFER
(return value of device_links_check_suppliers()).

However the semantics of DEVICE_LINK_SUPPLIER_UNBIND are "consumer not
bound and blocked from probing", with the above change it would become
"consumer may or may not be bound and blocked from probing".

Thus it would also be necessary to change device_links_driver_bound()
so that it doesn't update the status to DEVICE_LINK_ACTIVE.  Also,
device_links_busy() and device_links_unbind_consumers() would have
to check boundness with device_is_bound() if the status is
DEVICE_LINK_SUPPLIER_UNBIND.  Perhaps it would be easier to add
separate link states for this, or perhaps this problem is too
theoretical to bother dealing with it.

Thanks,

Lukas

Re: [Update][RFC/RFT][PATCH v3 2/5] driver core: Functional dependencies tracking support

[Rafael J. Wysocki]

On Wed, Sep 28, 2016 at 12:43 PM, Lukas Wunner <lukas@wunner.de> wrote:
> On Tue, Sep 27, 2016 at 01:52:48PM +0200, Rafael J. Wysocki wrote:
>> On Tue, Sep 27, 2016 at 10:54 AM, Lukas Wunner <lukas@wunner.de> wrote:
>> > On Fri, Sep 16, 2016 at 02:33:55PM +0200, Rafael J. Wysocki wrote:
>> >> +void device_links_unbind_consumers(struct device *dev)
>> >> +{
>> >> +     struct device_link *link;
>> >> +     int idx;
>> >> +
>> >> + start:
>> >> +     idx = device_links_read_lock();
>> >> +
>> >> +     list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
>> >> +             enum device_link_status status;
>> >> +
>> >> +             if (link->flags & DEVICE_LINK_STATELESS)
>> >> +                     continue;
>> >> +
>> >> +             spin_lock(&link->lock);
>> >> +             status = link->status;
>> >> +             if (status == DEVICE_LINK_CONSUMER_PROBE) {
>> >> +                     spin_unlock(&link->lock);
>> >> +
>> >> +                     device_links_read_unlock(idx);
>> >> +
>> >> +                     wait_for_device_probe();
>> >> +                     goto start;
>> >> +             }
>> >> +             link->status = DEVICE_LINK_SUPPLIER_UNBIND;
>> >
>> > While revisiting this function it just occurred to me that there's
>> > a theoretical infinite loop here if the consumer probes, is unbound
>> > by the supplier, then reprobes again before the supplier had a chance
>> > to update the link to DEVICE_LINK_SUPPLIER_UNBIND.  Perhaps this isn't
>> > a problem in practice, but noting anyway.
>>
>> But the consumer is unbound only after setting the link status to
>> DEVICE_LINK_SUPPLIER_UNBIND and then it won't probe again.
>
> Sorry, looking at the code with a fresh pair of eyeballs I realize the
> scenario for the infinite loop is different from what I've written above:
> The infinite loop can occur if the consumer probes continuously but never
> succeeds, e.g. due to some unfulfilled condition in its ->probe hook.

I'm not sure how that can happen.

If it doesn't succeed, the driver's ->probe() will return an error, so
that driver is not going to be tried again, unless the error is
-EPROBE_DEFER, but that will cause it to wait for another driver to
probe successfully in the meantime.

Or do you have any particular example in which things work differently in mind?

Thanks,
Rafael

Re: [Update][RFC/RFT][PATCH v3 2/5] driver core: Functional dependencies tracking support

[Lukas Wunner]

On Wed, Sep 28, 2016 at 01:31:36PM +0200, Rafael J. Wysocki wrote:
> On Wed, Sep 28, 2016 at 12:43 PM, Lukas Wunner <lukas@wunner.de> wrote:
> > On Tue, Sep 27, 2016 at 01:52:48PM +0200, Rafael J. Wysocki wrote:
> >> On Tue, Sep 27, 2016 at 10:54 AM, Lukas Wunner <lukas@wunner.de> wrote:
> >> > On Fri, Sep 16, 2016 at 02:33:55PM +0200, Rafael J. Wysocki wrote:
> >> >> +void device_links_unbind_consumers(struct device *dev)
> >> >> +{
> >> >> +     struct device_link *link;
> >> >> +     int idx;
> >> >> +
> >> >> + start:
> >> >> +     idx = device_links_read_lock();
> >> >> +
> >> >> +     list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
> >> >> +             enum device_link_status status;
> >> >> +
> >> >> +             if (link->flags & DEVICE_LINK_STATELESS)
> >> >> +                     continue;
> >> >> +
> >> >> +             spin_lock(&link->lock);
> >> >> +             status = link->status;
> >> >> +             if (status == DEVICE_LINK_CONSUMER_PROBE) {
> >> >> +                     spin_unlock(&link->lock);
> >> >> +
> >> >> +                     device_links_read_unlock(idx);
> >> >> +
> >> >> +                     wait_for_device_probe();
> >> >> +                     goto start;
> >> >> +             }
> >> >> +             link->status = DEVICE_LINK_SUPPLIER_UNBIND;
> >> >
> >> > While revisiting this function it just occurred to me that there's
> >> > a theoretical infinite loop here if the consumer probes, is unbound
> >> > by the supplier, then reprobes again before the supplier had a chance
> >> > to update the link to DEVICE_LINK_SUPPLIER_UNBIND.  Perhaps this isn't
> >> > a problem in practice, but noting anyway.
> >>
> >> But the consumer is unbound only after setting the link status to
> >> DEVICE_LINK_SUPPLIER_UNBIND and then it won't probe again.
> >
> > Sorry, looking at the code with a fresh pair of eyeballs I realize the
> > scenario for the infinite loop is different from what I've written above:
> > The infinite loop can occur if the consumer probes continuously but never
> > succeeds, e.g. due to some unfulfilled condition in its ->probe hook.
> 
> I'm not sure how that can happen.
> 
> If it doesn't succeed, the driver's ->probe() will return an error, so
> that driver is not going to be tried again, unless the error is
> -EPROBE_DEFER, but that will cause it to wait for another driver to
> probe successfully in the meantime.

You're right, it seems that the code is safe.  Sorry for the noise. :)

Best regards,

Lukas

[RFC/RFT][PATCH v3 3/5] PM / sleep: Make async suspend/resume of devices use device links

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Make the device suspend/resume part of the core system
suspend/resume code use device links to ensure that supplier
and consumer devices will be suspended and resumed in the right
order in case of async suspend/resume.

The idea, roughly, is to use dpm_wait() to wait for all consumers
before a supplier device suspend and to wait for all suppliers
before a consumer device resume.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/base/base.h       |    2 +
 drivers/base/core.c       |    4 +-
 drivers/base/power/main.c |   85 ++++++++++++++++++++++++++++++++++++++++++----
 3 files changed, 83 insertions(+), 8 deletions(-)

Index: linux-pm/drivers/base/base.h
===================================================================
--- linux-pm.orig/drivers/base/base.h
+++ linux-pm/drivers/base/base.h
@@ -163,3 +163,5 @@ extern void device_links_driver_gone(str
 extern void device_links_no_driver(struct device *dev);
 extern bool device_links_busy(struct device *dev);
 extern void device_links_unbind_consumers(struct device *dev);
+extern int device_links_read_lock(void);
+extern void device_links_read_unlock(int idx);
Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -212,12 +212,12 @@ void device_link_del(struct device_link
 }
 EXPORT_SYMBOL_GPL(device_link_del);
 
-static int device_links_read_lock(void)
+int device_links_read_lock(void)
 {
 	return srcu_read_lock(&device_links_srcu);
 }
 
-static void device_links_read_unlock(int idx)
+void device_links_read_unlock(int idx)
 {
 	return srcu_read_unlock(&device_links_srcu, idx);
 }
Index: linux-pm/drivers/base/power/main.c
===================================================================
--- linux-pm.orig/drivers/base/power/main.c
+++ linux-pm/drivers/base/power/main.c
@@ -244,6 +244,62 @@ static void dpm_wait_for_children(struct
        device_for_each_child(dev, &async, dpm_wait_fn);
 }
 
+static void dpm_wait_for_suppliers(struct device *dev, bool async)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	/*
+	 * If the supplier goes away right after we've checked the link to it,
+	 * we'll wait for its completion to change the state, but that's fine,
+	 * because the only things that will block as a result are the SRCU
+	 * callbacks freeing the link objects for the links in the list we're
+	 * walking.
+	 */
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node)
+		if (link->status != DEVICE_LINK_DORMANT)
+			dpm_wait(link->supplier, async);
+
+	device_links_read_unlock(idx);
+}
+
+static void dpm_wait_for_superior(struct device *dev, bool async)
+{
+	dpm_wait(dev->parent, async);
+	dpm_wait_for_suppliers(dev, async);
+}
+
+static void dpm_wait_for_consumers(struct device *dev, bool async)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	/*
+	 * The status of a device link can only be changed from "dormant" by a
+	 * probe, but that cannot happen during system suspend/resume.  In
+	 * theory it can change to "dormant" at that time, but then it is
+	 * reasonable to wait for the target device anyway (eg. if it goes
+	 * away, it's better to wait for it to go away completely and then
+	 * continue instead of trying to continue in parallel with its
+	 * unregistration).
+	 */
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node)
+		if (link->status != DEVICE_LINK_DORMANT)
+			dpm_wait(link->consumer, async);
+
+	device_links_read_unlock(idx);
+}
+
+static void dpm_wait_for_subordinate(struct device *dev, bool async)
+{
+	dpm_wait_for_children(dev, async);
+	dpm_wait_for_consumers(dev, async);
+}
+
 /**
  * pm_op - Return the PM operation appropriate for given PM event.
  * @ops: PM operations to choose from.
@@ -488,7 +544,7 @@ static int device_resume_noirq(struct de
 	if (!dev->power.is_noirq_suspended)
 		goto Out;
 
-	dpm_wait(dev->parent, async);
+	dpm_wait_for_superior(dev, async);
 
 	if (dev->pm_domain) {
 		info = "noirq power domain ";
@@ -618,7 +674,7 @@ static int device_resume_early(struct de
 	if (!dev->power.is_late_suspended)
 		goto Out;
 
-	dpm_wait(dev->parent, async);
+	dpm_wait_for_superior(dev, async);
 
 	if (dev->pm_domain) {
 		info = "early power domain ";
@@ -750,7 +806,7 @@ static int device_resume(struct device *
 		goto Complete;
 	}
 
-	dpm_wait(dev->parent, async);
+	dpm_wait_for_superior(dev, async);
 	dpm_watchdog_set(&wd, dev);
 	device_lock(dev);
 
@@ -1038,7 +1094,7 @@ static int __device_suspend_noirq(struct
 	if (dev->power.syscore || dev->power.direct_complete)
 		goto Complete;
 
-	dpm_wait_for_children(dev, async);
+	dpm_wait_for_subordinate(dev, async);
 
 	if (dev->pm_domain) {
 		info = "noirq power domain ";
@@ -1185,7 +1241,7 @@ static int __device_suspend_late(struct
 	if (dev->power.syscore || dev->power.direct_complete)
 		goto Complete;
 
-	dpm_wait_for_children(dev, async);
+	dpm_wait_for_subordinate(dev, async);
 
 	if (dev->pm_domain) {
 		info = "late power domain ";
@@ -1342,6 +1398,22 @@ static int legacy_suspend(struct device
 	return error;
 }
 
+static void dpm_clear_suppliers_direct_complete(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
+		spin_lock_irq(&link->supplier->power.lock);
+		link->supplier->power.direct_complete = false;
+		spin_unlock_irq(&link->supplier->power.lock);
+	}
+
+	device_links_read_unlock(idx);
+}
+
 /**
  * device_suspend - Execute "suspend" callbacks for given device.
  * @dev: Device to handle.
@@ -1358,7 +1430,7 @@ static int __device_suspend(struct devic
 	TRACE_DEVICE(dev);
 	TRACE_SUSPEND(0);
 
-	dpm_wait_for_children(dev, async);
+	dpm_wait_for_subordinate(dev, async);
 
 	if (async_error)
 		goto Complete;
@@ -1454,6 +1526,7 @@ static int __device_suspend(struct devic
 
 			spin_unlock_irq(&parent->power.lock);
 		}
+		dpm_clear_suppliers_direct_complete(dev);
 	}
 
 	device_unlock(dev);

[RFC/RFT][PATCH v3 4/5] PM / runtime: Use device links

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Modify the runtime PM framework to use device links to ensure that
supplier devices will not be suspended if any of their consumer
devices are active.

The idea is to reference count suppliers on the consumer's resume
and drop references to them on its suspend.  The information on
whether or not the supplier has been reference counted by the
consumer's (runtime) resume is stored in a new field (rpm_active)
in the link object for each link.

It may be necessary to clean up those references when the
supplier is unbinding and that's why the links whose status is
DEVICE_LINK_SUPPLIER_UNBIND are skipped by the runtime suspend
and resume code.

The above means that if the consumer device is probed in the
runtime-active state, the supplier has to be resumed and reference
counted by device_link_add() so the code works as expected on its
(runtime) suspend.  There is a new flag, DEVICE_LINK_RPM_ACTIVE,
to tell device_link_add() about that (in which case the caller
is responsible for making sure that the consumer really will
be runtime-active when runtime PM is enabled for it).

The other new link flag, DEVICE_LINK_PM_RUNTIME, tells the core
whether or not the link should be used for runtime PM at all.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/base/core.c          |   23 +++++++-
 drivers/base/dd.c            |    1 
 drivers/base/power/runtime.c |  121 +++++++++++++++++++++++++++++++++++++++++--
 include/linux/device.h       |    5 +
 include/linux/pm_runtime.h   |    2 
 5 files changed, 144 insertions(+), 8 deletions(-)

Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -101,8 +101,16 @@ static int device_reorder_to_tail(struct
  * @flags: Link flags.
  *
  * If the DEVICE_LINK_STATELESS flag is set, @status is ignored.  Otherwise,
- * the caller is responsible for ensuring that @status reflects the current
- * status of both @consumer and @supplier.
+ * the caller is responsible for ensuring that (a) @status reflects the current
+ * status of both @consumer and @supplier and (b) the creation of the link is
+ * properly synchronized with runtime PM.
+ *
+ * To that end, setting the DEVICE_LINK_PM_RUNTIME flag will cause the runtime
+ * PM framework to take the link into account.  Moreover, if the
+ * DEVICE_LINK_RPM_ACTIVE flag is set in addition to it, the supplier devices
+ * will be forced into the active metastate and reference-counted upon the
+ * creation of the link.  If DEVICE_LINK_PM_RUNTIME is not set,
+ * DEVICE_LINK_RPM_ACTIVE will be ignored.
  *
  * If the DEVICE_LINK_AUTOREMOVE is set, the link will be removed automatically
  * when the consumer device driver unbinds from it.  The combination of both
@@ -138,10 +146,19 @@ struct device_link *device_link_add(stru
 		if (link->consumer == consumer)
 			goto out;
 
-	link = kmalloc(sizeof(*link), GFP_KERNEL);
+	link = kzalloc(sizeof(*link), GFP_KERNEL);
 	if (!link)
 		goto out;
 
+	if ((flags & DEVICE_LINK_PM_RUNTIME) && (flags & DEVICE_LINK_RPM_ACTIVE)) {
+		if (pm_runtime_get_sync(supplier) < 0) {
+			pm_runtime_put_noidle(supplier);
+			kfree(link);
+			link = NULL;
+			goto out;
+		}
+		link->rpm_active = true;
+	}
 	get_device(supplier);
 	link->supplier = supplier;
 	INIT_LIST_HEAD(&link->s_node);
Index: linux-pm/drivers/base/dd.c
===================================================================
--- linux-pm.orig/drivers/base/dd.c
+++ linux-pm/drivers/base/dd.c
@@ -791,6 +791,7 @@ static void __device_release_driver(stru
 		}
 
 		pm_runtime_get_sync(dev);
+		pm_runtime_clean_up_links(dev);
 
 		driver_sysfs_remove(dev);
 
Index: linux-pm/drivers/base/power/runtime.c
===================================================================
--- linux-pm.orig/drivers/base/power/runtime.c
+++ linux-pm/drivers/base/power/runtime.c
@@ -12,6 +12,8 @@
 #include <linux/pm_runtime.h>
 #include <linux/pm_wakeirq.h>
 #include <trace/events/rpm.h>
+
+#include "../base.h"
 #include "power.h"
 
 typedef int (*pm_callback_t)(struct device *);
@@ -258,6 +260,42 @@ static int rpm_check_suspend_allowed(str
 	return retval;
 }
 
+static int rpm_get_suppliers(struct device *dev)
+{
+	struct device_link *link;
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
+		int retval;
+
+		if (!(link->flags & DEVICE_LINK_PM_RUNTIME))
+			continue;
+
+		if (link->status == DEVICE_LINK_SUPPLIER_UNBIND ||
+		    link->rpm_active)
+			continue;
+
+		retval = pm_runtime_get_sync(link->supplier);
+		if (retval < 0) {
+			pm_runtime_put_noidle(link->supplier);
+			return retval;
+		}
+		link->rpm_active = true;
+	}
+	return 0;
+}
+
+static void rpm_put_suppliers(struct device *dev)
+{
+	struct device_link *link;
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node)
+		if (link->rpm_active &&
+		    link->status != DEVICE_LINK_SUPPLIER_UNBIND) {
+			pm_runtime_put(link->supplier);
+			link->rpm_active = false;
+		}
+}
+
 /**
  * __rpm_callback - Run a given runtime PM callback for a given device.
  * @cb: Runtime PM callback to run.
@@ -266,19 +304,55 @@ static int rpm_check_suspend_allowed(str
 static int __rpm_callback(int (*cb)(struct device *), struct device *dev)
 	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
-	int retval;
+	int retval, idx;
 
-	if (dev->power.irq_safe)
+	if (dev->power.irq_safe) {
 		spin_unlock(&dev->power.lock);
-	else
+	} else {
 		spin_unlock_irq(&dev->power.lock);
 
+		/*
+		 * Resume suppliers if necessary.
+		 *
+		 * The device's runtime PM status cannot change until this
+		 * routine returns, so it is safe to read the status outside of
+		 * the lock.
+		 */
+		if (dev->power.runtime_status == RPM_RESUMING) {
+			idx = device_links_read_lock();
+
+			retval = rpm_get_suppliers(dev);
+			if (retval)
+				goto fail;
+
+			device_links_read_unlock(idx);
+		}
+	}
+
 	retval = cb(dev);
 
-	if (dev->power.irq_safe)
+	if (dev->power.irq_safe) {
 		spin_lock(&dev->power.lock);
-	else
+	} else {
+		/*
+		 * If the device is suspending and the callback has returned
+		 * success, drop the usage counters of the suppliers that have
+		 * been reference counted on its resume.
+		 *
+		 * Do that if resume fails too.
+		 */
+		if ((dev->power.runtime_status == RPM_SUSPENDING && !retval)
+		    || (dev->power.runtime_status == RPM_RESUMING && retval)) {
+			idx = device_links_read_lock();
+
+ fail:
+			rpm_put_suppliers(dev);
+
+			device_links_read_unlock(idx);
+		}
+
 		spin_lock_irq(&dev->power.lock);
+	}
 
 	return retval;
 }
@@ -1447,6 +1521,43 @@ void pm_runtime_remove(struct device *de
 }
 
 /**
+ * pm_runtime_clean_up_links - Prepare links to consumers for driver removal.
+ * @dev: Device whose driver is going to be removed.
+ *
+ * Check links from this device to any consumers and if any of them have active
+ * runtime PM references to the device, drop the usage counter of the device
+ * (once per link).
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ *
+ * Since the device is guaranteed to be runtime-active at the point this is
+ * called, nothing else needs to be done here.
+ *
+ * Moreover, this is called after device_links_busy() has returned 'false', so
+ * the status of each link is guaranteed to be DEVICE_LINK_SUPPLIER_UNBIND and
+ * therefore rpm_active can't be manipulated concurrently.
+ */
+void pm_runtime_clean_up_links(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		if (link->rpm_active) {
+			pm_runtime_put_noidle(dev);
+			link->rpm_active = false;
+		}
+	}
+
+	device_links_read_unlock(idx);
+}
+
+/**
  * pm_runtime_force_suspend - Force a device into suspend state if needed.
  * @dev: Device to suspend.
  *
Index: linux-pm/include/linux/device.h
===================================================================
--- linux-pm.orig/include/linux/device.h
+++ linux-pm/include/linux/device.h
@@ -720,9 +720,13 @@ enum device_link_status {
  *
  * STATELESS: The state machine is not applicable to this link.
  * AUTOREMOVE: Remove this link automatically on cunsumer driver unbind.
+ * PM_RUNTIME: If set, the runtime PM framework will use this link.
+ * RPM_ACTIVE: Run pm_runtime_get_sync() on the supplier during link creation.
  */
 #define DEVICE_LINK_STATELESS	(1 << 0)
 #define DEVICE_LINK_AUTOREMOVE	(1 << 1)
+#define DEVICE_LINK_PM_RUNTIME	(1 << 2)
+#define DEVICE_LINK_RPM_ACTIVE	(1 << 3)
 
 struct device_link {
 	struct device *supplier;
@@ -731,6 +735,7 @@ struct device_link {
 	struct list_head c_node;
 	enum device_link_status status;
 	u32 flags;
+	bool rpm_active;
 	spinlock_t lock;
 	struct rcu_head rcu_head;
 };
Index: linux-pm/include/linux/pm_runtime.h
===================================================================
--- linux-pm.orig/include/linux/pm_runtime.h
+++ linux-pm/include/linux/pm_runtime.h
@@ -55,6 +55,7 @@ extern unsigned long pm_runtime_autosusp
 extern void pm_runtime_update_max_time_suspended(struct device *dev,
 						 s64 delta_ns);
 extern void pm_runtime_set_memalloc_noio(struct device *dev, bool enable);
+extern void pm_runtime_clean_up_links(struct device *dev);
 
 static inline void pm_suspend_ignore_children(struct device *dev, bool enable)
 {
@@ -186,6 +187,7 @@ static inline unsigned long pm_runtime_a
 				struct device *dev) { return 0; }
 static inline void pm_runtime_set_memalloc_noio(struct device *dev,
 						bool enable){}
+static inline void pm_runtime_clean_up_links(struct device *dev) {}
 
 #endif /* !CONFIG_PM */
 

[RFC/RFT][PATCH v3 5/5] PM / runtime: Optimize the use of device links

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

If the device has no links to suppliers that should be used for
runtime PM (links with DEVICE_LINK_PM_RUNTIME set), there is no
reason to walk the list of suppliers for that device during
runtime suspend and resume.

Add a simple mechanism to detect that case and possibly avoid the
extra unnecessary overhead.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/base/core.c          |   20 +++++++++++++-------
 drivers/base/power/runtime.c |   23 ++++++++++++++++++++---
 include/linux/pm.h           |    1 +
 include/linux/pm_runtime.h   |    4 ++++
 4 files changed, 38 insertions(+), 10 deletions(-)

Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -150,14 +150,17 @@ struct device_link *device_link_add(stru
 	if (!link)
 		goto out;
 
-	if ((flags & DEVICE_LINK_PM_RUNTIME) && (flags & DEVICE_LINK_RPM_ACTIVE)) {
-		if (pm_runtime_get_sync(supplier) < 0) {
-			pm_runtime_put_noidle(supplier);
-			kfree(link);
-			link = NULL;
-			goto out;
+	if (flags & DEVICE_LINK_PM_RUNTIME) {
+		if (flags & DEVICE_LINK_RPM_ACTIVE) {
+			if (pm_runtime_get_sync(supplier) < 0) {
+				pm_runtime_put_noidle(supplier);
+				kfree(link);
+				link = NULL;
+				goto out;
+			}
+			link->rpm_active = true;
 		}
-		link->rpm_active = true;
+		pm_runtime_new_link(consumer);
 	}
 	get_device(supplier);
 	link->supplier = supplier;
@@ -207,6 +210,9 @@ static void __device_link_del(struct dev
 	dev_info(link->consumer, "Dropping the link to %s\n",
 		 dev_name(link->supplier));
 
+	if (link->flags & DEVICE_LINK_PM_RUNTIME)
+		pm_runtime_drop_link(link->consumer);
+
 	list_del_rcu(&link->s_node);
 	list_del_rcu(&link->c_node);
 	call_srcu(&device_links_srcu, &link->rcu_head, __device_link_free_srcu);
Index: linux-pm/drivers/base/power/runtime.c
===================================================================
--- linux-pm.orig/drivers/base/power/runtime.c
+++ linux-pm/drivers/base/power/runtime.c
@@ -305,6 +305,7 @@ static int __rpm_callback(int (*cb)(stru
 	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
 	int retval, idx;
+	bool use_links = dev->power.links_count > 0;
 
 	if (dev->power.irq_safe) {
 		spin_unlock(&dev->power.lock);
@@ -318,7 +319,7 @@ static int __rpm_callback(int (*cb)(stru
 		 * routine returns, so it is safe to read the status outside of
 		 * the lock.
 		 */
-		if (dev->power.runtime_status == RPM_RESUMING) {
+		if (use_links && dev->power.runtime_status == RPM_RESUMING) {
 			idx = device_links_read_lock();
 
 			retval = rpm_get_suppliers(dev);
@@ -341,8 +342,9 @@ static int __rpm_callback(int (*cb)(stru
 		 *
 		 * Do that if resume fails too.
 		 */
-		if ((dev->power.runtime_status == RPM_SUSPENDING && !retval)
-		    || (dev->power.runtime_status == RPM_RESUMING && retval)) {
+		if (use_links
+		    && ((dev->power.runtime_status == RPM_SUSPENDING && !retval)
+		    || (dev->power.runtime_status == RPM_RESUMING && retval))) {
 			idx = device_links_read_lock();
 
  fail:
@@ -1557,6 +1559,21 @@ void pm_runtime_clean_up_links(struct de
 	device_links_read_unlock(idx);
 }
 
+void pm_runtime_new_link(struct device *dev)
+{
+	spin_lock_irq(&dev->power.lock);
+	dev->power.links_count++;
+	spin_unlock_irq(&dev->power.lock);
+}
+
+void pm_runtime_drop_link(struct device *dev)
+{
+	spin_lock_irq(&dev->power.lock);
+	WARN_ON(dev->power.links_count == 0);
+	dev->power.links_count--;
+	spin_unlock_irq(&dev->power.lock);
+}
+
 /**
  * pm_runtime_force_suspend - Force a device into suspend state if needed.
  * @dev: Device to suspend.
Index: linux-pm/include/linux/pm.h
===================================================================
--- linux-pm.orig/include/linux/pm.h
+++ linux-pm/include/linux/pm.h
@@ -596,6 +596,7 @@ struct dev_pm_info {
 	unsigned int		use_autosuspend:1;
 	unsigned int		timer_autosuspends:1;
 	unsigned int		memalloc_noio:1;
+	unsigned int		links_count;
 	enum rpm_request	request;
 	enum rpm_status		runtime_status;
 	int			runtime_error;
Index: linux-pm/include/linux/pm_runtime.h
===================================================================
--- linux-pm.orig/include/linux/pm_runtime.h
+++ linux-pm/include/linux/pm_runtime.h
@@ -56,6 +56,8 @@ extern void pm_runtime_update_max_time_s
 						 s64 delta_ns);
 extern void pm_runtime_set_memalloc_noio(struct device *dev, bool enable);
 extern void pm_runtime_clean_up_links(struct device *dev);
+extern void pm_runtime_new_link(struct device *dev);
+extern void pm_runtime_drop_link(struct device *dev);
 
 static inline void pm_suspend_ignore_children(struct device *dev, bool enable)
 {
@@ -188,6 +190,8 @@ static inline unsigned long pm_runtime_a
 static inline void pm_runtime_set_memalloc_noio(struct device *dev,
 						bool enable){}
 static inline void pm_runtime_clean_up_links(struct device *dev) {}
+static inline void pm_runtime_new_link(struct device *dev) {}
+static inline void pm_runtime_drop_link(struct device *dev) {}
 
 #endif /* !CONFIG_PM */
 

Re: [RFC/RFT][PATCH v3 0/5] Functional dependencies between devices

[Marek Szyprowski]

Hi Rafael,


On 2016-09-16 00:03, Rafael J. Wysocki wrote:
> Hi Everyone,
>
> On Thursday, September 08, 2016 11:25:44 PM Rafael J. Wysocki wrote:
>> Hi Everyone,
>>
>> This is a refresh of the functional dependencies series that I posted last
>> year and which has picked up by Marek quite recently.  For reference, appended
>> is my introductory message sent previously (which may be slightly outdated now).
>>
>> As last time, the first patch rearranges the code around __device_release_driver()
>> a bit to prepare it for the next one (it actually hasn't changed AFAICS).
>>
>> The second patch introduces the actual device links mechanics, but without
>> system suspend/resume and runtime PM support which are added by the subsequent
>> patches.
>>
>> Some bugs found by Marek during his work on these patches should be fixed
>> here.  In particular, the endless recursion in device_reorder_to_tail()
>> which simply was broken before.
>>
>> There are two additional patches to address the issue with runtime PM support
>> that occured when runtime PM was disabled for some suppliers due to a PM
>> sleep transition in progress.  Those patches simply make runtime PM helpers
>> return 0 in that case which may be controversial, so please let me know if
>> there are concerns about those.
>>
>> The way device_link_add() works is a bit different, as it takes an additional
>> status argument now.  That makes it possible to create a link in any state,
>> with extra care of course, and should address the problem pointed to by Lukas
>> during the previous discussion.
>>
>> Also some comments from Tomeu have been addressed.
> An update here.
>
> The first patch hasn't changed, so I'm resending it.
>
> The majority of changes in the other patches are in order to address Lukas'
> comments.
>
> First off, I added a DEVICE_LINK_STATELESS flag that will prevent the driver
> core from trying to maintain device links having it set.
>
> Also, the DEVICE_LINK_PERSISTENT flag was dropped (as link "persistence" is the
> default behavior now) and there's a new one, DEVICE_LINK_AUTOREMOVE, that will
> cause the driver core to remove the link on the consumer driver unbind.
>
> Moreover, the code checks attempts to create a link between a parent and a child
> device now and actively prevents that from happening.
>
> The changelog of the second patch has been updated as requested by Ulf.
>
> The third patch was updated to fix a bug related to the (previously missing)
> clearing of power.direct_complete for supplier devices having consumers that
> don't use direct_complete.
>
> The next two (runtime PM) patches turned out to be unnecessary, so I've dropped
> them.
>
> The runtime PM patch [4/5] was reorganized somewhat to reduce the indentation
> level in there, but the code flow introduced by it is essentially the same
> and the last patch was simply rebased on top of the new series.
>
> If this version still works for Marek, I'll probably drop the RFC tag from it
> in the next iteration.

Sadly, this version doesn't work. I get following kernel bug:

[    2.357622] BUG: spinlock bad magic on CPU#0, swapper/0/1
[    2.362361]  lock: 0xeea2e294, .magic: ffffffff, .owner: /0, 
.owner_cpu: -1
[    2.369389] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 
4.8.0-rc6-00019-gd66d0028dd3c-dirty #651
[    2.377954] Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
[    2.384053] [<c010d7f0>] (unwind_backtrace) from [<c010a4b4>] 
(show_stack+0x10/0x14)
[    2.391766] [<c010a4b4>] (show_stack) from [<c032220c>] 
(dump_stack+0x74/0x94)
[    2.398970] [<c032220c>] (dump_stack) from [<c0158e8c>] 
(do_raw_spin_lock+0x160/0x1a8)
[    2.406870] [<c0158e8c>] (do_raw_spin_lock) from [<c03e8d84>] 
(device_links_no_driver+0x64/0x98)
[    2.415634] [<c03e8d84>] (device_links_no_driver) from [<c03ec32c>] 
(driver_probe_device+0xa0/0x2bc)
[    2.424744] [<c03ec32c>] (driver_probe_device) from [<c03ec5f4>] 
(__driver_attach+0xac/0xb0)
[    2.433165] [<c03ec5f4>] (__driver_attach) from [<c03eaa90>] 
(bus_for_each_dev+0x54/0x88)
[    2.441323] [<c03eaa90>] (bus_for_each_dev) from [<c03eba6c>] 
(bus_add_driver+0xe8/0x1f4)
[    2.449481] [<c03eba6c>] (bus_add_driver) from [<c03ece54>] 
(driver_register+0x78/0xf4)
[    2.457469] [<c03ece54>] (driver_register) from [<c010178c>] 
(do_one_initcall+0x3c/0x16c)
[    2.465632] [<c010178c>] (do_one_initcall) from [<c0b00d84>] 
(kernel_init_freeable+0x120/0x1ec)
[    2.474313] [<c0b00d84>] (kernel_init_freeable) from [<c0704194>] 
(kernel_init+0x8/0x118)
[    2.482470] [<c0704194>] (kernel_init) from [<c01079b8>] 
(ret_from_fork+0x14/0x3c)

I'm checking what's wrong there.

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [RFC/RFT][PATCH v3 0/5] Functional dependencies between devices

[Marek Szyprowski]

Hi Everyone,

On 2016-09-16 09:25, Marek Szyprowski wrote:
> Hi Rafael,
>
> On 2016-09-16 00:03, Rafael J. Wysocki wrote:
>> Hi Everyone,
>>
>> On Thursday, September 08, 2016 11:25:44 PM Rafael J. Wysocki wrote:
>>> Hi Everyone,
>>>
>>> This is a refresh of the functional dependencies series that I 
>>> posted last
>>> year and which has picked up by Marek quite recently.  For 
>>> reference, appended
>>> is my introductory message sent previously (which may be slightly 
>>> outdated now).
>>>
>>> As last time, the first patch rearranges the code around 
>>> __device_release_driver()
>>> a bit to prepare it for the next one (it actually hasn't changed 
>>> AFAICS).
>>>
>>> The second patch introduces the actual device links mechanics, but 
>>> without
>>> system suspend/resume and runtime PM support which are added by the 
>>> subsequent
>>> patches.
>>>
>>> Some bugs found by Marek during his work on these patches should be 
>>> fixed
>>> here.  In particular, the endless recursion in device_reorder_to_tail()
>>> which simply was broken before.
>>>
>>> There are two additional patches to address the issue with runtime 
>>> PM support
>>> that occured when runtime PM was disabled for some suppliers due to 
>>> a PM
>>> sleep transition in progress.  Those patches simply make runtime PM 
>>> helpers
>>> return 0 in that case which may be controversial, so please let me 
>>> know if
>>> there are concerns about those.
>>>
>>> The way device_link_add() works is a bit different, as it takes an 
>>> additional
>>> status argument now.  That makes it possible to create a link in any 
>>> state,
>>> with extra care of course, and should address the problem pointed to 
>>> by Lukas
>>> during the previous discussion.
>>>
>>> Also some comments from Tomeu have been addressed.
>> An update here.
>>
>> The first patch hasn't changed, so I'm resending it.
>>
>> The majority of changes in the other patches are in order to address 
>> Lukas'
>> comments.
>>
>> First off, I added a DEVICE_LINK_STATELESS flag that will prevent the 
>> driver
>> core from trying to maintain device links having it set.
>>
>> Also, the DEVICE_LINK_PERSISTENT flag was dropped (as link 
>> "persistence" is the
>> default behavior now) and there's a new one, DEVICE_LINK_AUTOREMOVE, 
>> that will
>> cause the driver core to remove the link on the consumer driver unbind.
>>
>> Moreover, the code checks attempts to create a link between a parent 
>> and a child
>> device now and actively prevents that from happening.
>>
>> The changelog of the second patch has been updated as requested by Ulf.
>>
>> The third patch was updated to fix a bug related to the (previously 
>> missing)
>> clearing of power.direct_complete for supplier devices having 
>> consumers that
>> don't use direct_complete.
>>
>> The next two (runtime PM) patches turned out to be unnecessary, so 
>> I've dropped
>> them.
>>
>> The runtime PM patch [4/5] was reorganized somewhat to reduce the 
>> indentation
>> level in there, but the code flow introduced by it is essentially the 
>> same
>> and the last patch was simply rebased on top of the new series.
>>
>> If this version still works for Marek, I'll probably drop the RFC tag 
>> from it
>> in the next iteration.
>
> Sadly, this version doesn't work. I get following kernel bug:
>
> [    2.357622] BUG: spinlock bad magic on CPU#0, swapper/0/1
> [    2.362361]  lock: 0xeea2e294, .magic: ffffffff, .owner: /0, 
> .owner_cpu: -1
> [    2.369389] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 
> 4.8.0-rc6-00019-gd66d0028dd3c-dirty #651
> [    2.377954] Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
> [    2.384053] [<c010d7f0>] (unwind_backtrace) from [<c010a4b4>] 
> (show_stack+0x10/0x14)
> [    2.391766] [<c010a4b4>] (show_stack) from [<c032220c>] 
> (dump_stack+0x74/0x94)
> [    2.398970] [<c032220c>] (dump_stack) from [<c0158e8c>] 
> (do_raw_spin_lock+0x160/0x1a8)
> [    2.406870] [<c0158e8c>] (do_raw_spin_lock) from [<c03e8d84>] 
> (device_links_no_driver+0x64/0x98)
> [    2.415634] [<c03e8d84>] (device_links_no_driver) from [<c03ec32c>] 
> (driver_probe_device+0xa0/0x2bc)
> [    2.424744] [<c03ec32c>] (driver_probe_device) from [<c03ec5f4>] 
> (__driver_attach+0xac/0xb0)
> [    2.433165] [<c03ec5f4>] (__driver_attach) from [<c03eaa90>] 
> (bus_for_each_dev+0x54/0x88)
> [    2.441323] [<c03eaa90>] (bus_for_each_dev) from [<c03eba6c>] 
> (bus_add_driver+0xe8/0x1f4)
> [    2.449481] [<c03eba6c>] (bus_add_driver) from [<c03ece54>] 
> (driver_register+0x78/0xf4)
> [    2.457469] [<c03ece54>] (driver_register) from [<c010178c>] 
> (do_one_initcall+0x3c/0x16c)
> [    2.465632] [<c010178c>] (do_one_initcall) from [<c0b00d84>] 
> (kernel_init_freeable+0x120/0x1ec)
> [    2.474313] [<c0b00d84>] (kernel_init_freeable) from [<c0704194>] 
> (kernel_init+0x8/0x118)
> [    2.482470] [<c0704194>] (kernel_init) from [<c01079b8>] 
> (ret_from_fork+0x14/0x3c)
>
> I'm checking what's wrong there.
>

The issue was caused by missing braces in device_links_no_driver() 
function.
After fixing it the patches works fine, so you can add:

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>

Rafael, how and when do you plan to merge them? I would like to know how to
process further with my IOMMU patch, which is depends on your patches.

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [RFC/RFT][PATCH v3 0/5] Functional dependencies between devices

[Rafael J. Wysocki]

On Fri, Sep 16, 2016 at 9:57 AM, Marek Szyprowski
<m.szyprowski@samsung.com> wrote:
> Hi Everyone,
>
>
> On 2016-09-16 09:25, Marek Szyprowski wrote:
>>
>> Hi Rafael,
>>
>> On 2016-09-16 00:03, Rafael J. Wysocki wrote:
>>>
>>> Hi Everyone,
>>>
>>> On Thursday, September 08, 2016 11:25:44 PM Rafael J. Wysocki wrote:
>>>>
>>>> Hi Everyone,
>>>>
>>>> This is a refresh of the functional dependencies series that I posted
>>>> last
>>>> year and which has picked up by Marek quite recently.  For reference,
>>>> appended
>>>> is my introductory message sent previously (which may be slightly
>>>> outdated now).
>>>>
>>>> As last time, the first patch rearranges the code around
>>>> __device_release_driver()
>>>> a bit to prepare it for the next one (it actually hasn't changed
>>>> AFAICS).
>>>>
>>>> The second patch introduces the actual device links mechanics, but
>>>> without
>>>> system suspend/resume and runtime PM support which are added by the
>>>> subsequent
>>>> patches.
>>>>
>>>> Some bugs found by Marek during his work on these patches should be
>>>> fixed
>>>> here.  In particular, the endless recursion in device_reorder_to_tail()
>>>> which simply was broken before.
>>>>
>>>> There are two additional patches to address the issue with runtime PM
>>>> support
>>>> that occured when runtime PM was disabled for some suppliers due to a PM
>>>> sleep transition in progress.  Those patches simply make runtime PM
>>>> helpers
>>>> return 0 in that case which may be controversial, so please let me know
>>>> if
>>>> there are concerns about those.
>>>>
>>>> The way device_link_add() works is a bit different, as it takes an
>>>> additional
>>>> status argument now.  That makes it possible to create a link in any
>>>> state,
>>>> with extra care of course, and should address the problem pointed to by
>>>> Lukas
>>>> during the previous discussion.
>>>>
>>>> Also some comments from Tomeu have been addressed.
>>>
>>> An update here.
>>>
>>> The first patch hasn't changed, so I'm resending it.
>>>
>>> The majority of changes in the other patches are in order to address
>>> Lukas'
>>> comments.
>>>
>>> First off, I added a DEVICE_LINK_STATELESS flag that will prevent the
>>> driver
>>> core from trying to maintain device links having it set.
>>>
>>> Also, the DEVICE_LINK_PERSISTENT flag was dropped (as link "persistence"
>>> is the
>>> default behavior now) and there's a new one, DEVICE_LINK_AUTOREMOVE, that
>>> will
>>> cause the driver core to remove the link on the consumer driver unbind.
>>>
>>> Moreover, the code checks attempts to create a link between a parent and
>>> a child
>>> device now and actively prevents that from happening.
>>>
>>> The changelog of the second patch has been updated as requested by Ulf.
>>>
>>> The third patch was updated to fix a bug related to the (previously
>>> missing)
>>> clearing of power.direct_complete for supplier devices having consumers
>>> that
>>> don't use direct_complete.
>>>
>>> The next two (runtime PM) patches turned out to be unnecessary, so I've
>>> dropped
>>> them.
>>>
>>> The runtime PM patch [4/5] was reorganized somewhat to reduce the
>>> indentation
>>> level in there, but the code flow introduced by it is essentially the
>>> same
>>> and the last patch was simply rebased on top of the new series.
>>>
>>> If this version still works for Marek, I'll probably drop the RFC tag
>>> from it
>>> in the next iteration.
>>
>>
>> Sadly, this version doesn't work. I get following kernel bug:
>>
>> [    2.357622] BUG: spinlock bad magic on CPU#0, swapper/0/1
>> [    2.362361]  lock: 0xeea2e294, .magic: ffffffff, .owner: /0,
>> .owner_cpu: -1
>> [    2.369389] CPU: 0 PID: 1 Comm: swapper/0 Not tainted
>> 4.8.0-rc6-00019-gd66d0028dd3c-dirty #651
>> [    2.377954] Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
>> [    2.384053] [<c010d7f0>] (unwind_backtrace) from [<c010a4b4>]
>> (show_stack+0x10/0x14)
>> [    2.391766] [<c010a4b4>] (show_stack) from [<c032220c>]
>> (dump_stack+0x74/0x94)
>> [    2.398970] [<c032220c>] (dump_stack) from [<c0158e8c>]
>> (do_raw_spin_lock+0x160/0x1a8)
>> [    2.406870] [<c0158e8c>] (do_raw_spin_lock) from [<c03e8d84>]
>> (device_links_no_driver+0x64/0x98)
>> [    2.415634] [<c03e8d84>] (device_links_no_driver) from [<c03ec32c>]
>> (driver_probe_device+0xa0/0x2bc)
>> [    2.424744] [<c03ec32c>] (driver_probe_device) from [<c03ec5f4>]
>> (__driver_attach+0xac/0xb0)
>> [    2.433165] [<c03ec5f4>] (__driver_attach) from [<c03eaa90>]
>> (bus_for_each_dev+0x54/0x88)
>> [    2.441323] [<c03eaa90>] (bus_for_each_dev) from [<c03eba6c>]
>> (bus_add_driver+0xe8/0x1f4)
>> [    2.449481] [<c03eba6c>] (bus_add_driver) from [<c03ece54>]
>> (driver_register+0x78/0xf4)
>> [    2.457469] [<c03ece54>] (driver_register) from [<c010178c>]
>> (do_one_initcall+0x3c/0x16c)
>> [    2.465632] [<c010178c>] (do_one_initcall) from [<c0b00d84>]
>> (kernel_init_freeable+0x120/0x1ec)
>> [    2.474313] [<c0b00d84>] (kernel_init_freeable) from [<c0704194>]
>> (kernel_init+0x8/0x118)
>> [    2.482470] [<c0704194>] (kernel_init) from [<c01079b8>]
>> (ret_from_fork+0x14/0x3c)
>>
>> I'm checking what's wrong there.
>>
>
> The issue was caused by missing braces in device_links_no_driver() function.

I'll send an update of the patch in question shortly.

> After fixing it the patches works fine, so you can add:
>
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>

Thanks!

> Rafael, how and when do you plan to merge them? I would like to know how to
> process further with my IOMMU patch, which is depends on your patches.

The "how" part really depends on Greg.  Nothing in my queue depends on
these patches at the moment, so I have no hard preferences.

As far as the "when" part goes, realistically, we are about a week
away from the 4.9 merge window I think, so 4.10 would be my target.

Thanks,
Rafael

Re: [RFC/RFT][PATCH v3 0/5] Functional dependencies between devices

[Lukas Wunner]

[+cc corbet]

To whom it may concern,

I made some notes while reviewing the state machine in patch 2 of this
series and thought, why not rework it into something that could eventually
go into the Documentation/ tree?

So here's an initial draft.  There's some introductory text plus
a description of the state machine.  Just putting this out there now
to ease reviewers' lives, despite the obvious WIP status.  I'll try to
amend it as the series converges.

This is already rst-formatted but I haven't actually run it through
sphinx yet.

Lukas

-- >8 --

Subject: [PATCH] Documentation: device links: Add initial documentation

Signed-off-by: Lukas Wunner <lukas@wunner.de>
---
 Documentation/driver-model/device_link.rst | 95 ++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
 create mode 100644 Documentation/driver-model/device_link.rst

diff --git a/Documentation/driver-model/device_link.rst b/Documentation/driver-model/device_link.rst
new file mode 100644
index 0000000..ba911b4
--- /dev/null
+++ b/Documentation/driver-model/device_link.rst
@@ -0,0 +1,95 @@
+============
+Device links
+============
+
+By default, the driver core only enforces dependencies between devices
+that are borne out of a parent/child relationship within the device
+hierarchy: When suspending, resuming or shutting down the system, devices
+are ordered based on this relationship, i.e. children are always suspended
+before their parent, and the parent is always resumed before its children.
+
+Sometimes there is a need to represent device dependencies beyond the
+mere parent/child relationship, e.g. between siblings, and have the
+driver core automatically take care of them.
+
+Secondly, the driver core by default does not enforce any driver presence
+dependencies, i.e. that one device must be bound to a driver before
+another one can probe or function correctly.
+
+Often these two dependency types come together, so a device depends on
+another one both with regards to driver presence *and* with regards to
+suspend/resume and shutdown ordering.
+
+Device links allow representation of such dependencies in the driver core.
+
+In its standard form, a device link combines *both* dependency types:
+It guarantees correct suspend/resume and shutdown ordering between a
+"supplier" device and its "consumer" devices, and it guarantees driver
+presence on the supplier:  The consumer devices are not probed before the
+supplier is bound to a driver, and they're unbound before the supplier
+is unbound.
+
+When driver presence on the supplier is irrelevant and only correct
+suspend/resume and shutdown ordering is needed, the device link may
+simply be set up with the DEVICE_LINK_STATELESS flag.
+
+A driver presence dependency between parent and child, i.e. within the
+regular device hierarchy, could in principle also be represented in the
+driver core using a device link.
+
+If a device link is set up with the DEVICE_LINK_AUTOREMOVE flag, it is
+automatically purged when the consumer fails to probe or later unbinds.
+This is handy when adding a device link from the consumer's ->probe hook,
+as it obviates the need to delete the link in the ->remove hook or in
+the error path of the ->probe hook.
+
+
+State machine
+=============
+
+"""
+                .=============================.
+                |                             |
+                v                             |
+DORMANT <=> AVAILABLE <=> CONSUMER_PROBE => ACTIVE
+   ^                                          |
+   |                                          |
+   '============ SUPPLIER_UNBIND <============'
+"""
+
+* The initial state of a device link is passed in to device_link_add().
+  If the link is created before any devices are probed, it must be set to
+  DEVICE_LINK_DORMANT.
+
+* When a supplier device is bound to a driver, links to its consumers
+  progress to DEVICE_LINK_AVAILABLE.
+  (Call to device_links_driver_bound() from driver_bound().)
+
+* Before a consumer device is probed, presence of supplier drivers is
+  verified by checking that links to suppliers are in DEVICE_LINK_AVAILABLE
+  state.  The state of the links is updated to DEVICE_LINK_CONSUMER_PROBE.
+  (Call to device_links_check_suppliers() from driver_probe_device().)
+  This prevents the supplier from unbinding.
+  (Call to wait_for_device_probe() in device_links_unbind_consumers().)
+
+* If the probe fails, links to suppliers revert back to DEVICE_LINK_AVAILABLE.
+  (Call to device_links_no_driver() from really_probe().)
+
+* If the probe succeeds, links to suppliers progress to DEVICE_LINK_ACTIVE.
+  (Call of device_links_driver_bound() from driver_bound().)
+
+* When the consumer's driver is later on removed, links to suppliers revert
+  back to DEVICE_LINK_AVAILABLE.
+  (Call to device_links_no_driver() from __device_release_driver().)
+
+* Before a supplier's driver is removed, links to consumers that are not
+  bound to a driver are updated to DEVICE_LINK_SUPPLIER_UNBIND.
+  (Call to device_links_busy() from __device_release_driver().)
+  This prevents the consumers from binding.
+  (Call to device_links_check_suppliers() from driver_probe_device().)
+  Consumers that are bound are freed from their driver; consumers that are
+  probing are waited for until they are done.
+  (Call to device_links_unbind_consumers() from __device_release_driver().)
+  Once all links to consumers are in DEVICE_LINK_SUPPLIER_UNBIND state,
+  the supplier driver is released and the links revert to DEVICE_LINK_DORMANT.
+  (Call to device_links_driver_gone() from __device_release_driver().)
-- 
2.9.3

Re: [RFC/RFT][PATCH v3 0/5] Functional dependencies between devices

[Rafael J. Wysocki]

On Tuesday, September 27, 2016 02:34:29 PM Lukas Wunner wrote:
> [+cc corbet]
> 
> To whom it may concern,
> 
> I made some notes while reviewing the state machine in patch 2 of this
> series and thought, why not rework it into something that could eventually
> go into the Documentation/ tree?
> 
> So here's an initial draft.  There's some introductory text plus
> a description of the state machine.  Just putting this out there now
> to ease reviewers' lives, despite the obvious WIP status.  I'll try to
> amend it as the series converges.
> 
> This is already rst-formatted but I haven't actually run it through
> sphinx yet.

Thanks a lot for doing this!

It looks good to me in general.  I think it would be good to add it to the
series at one point (if you don't mind).

I'm only a bit reluctant about advertising the usage of links between children
and parents, because that doesn't look like the right tool for the purpose
(as I said before, I'd prefer to add a device flag causing the parent driver
to be probed before the child one if needed).

Thanks,
Rafael


> -- >8 --
> 
> Subject: [PATCH] Documentation: device links: Add initial documentation
> 
> Signed-off-by: Lukas Wunner <lukas@wunner.de>
> ---
>  Documentation/driver-model/device_link.rst | 95 ++++++++++++++++++++++++++++++
>  1 file changed, 95 insertions(+)
>  create mode 100644 Documentation/driver-model/device_link.rst
> 
> diff --git a/Documentation/driver-model/device_link.rst b/Documentation/driver-model/device_link.rst
> new file mode 100644
> index 0000000..ba911b4
> --- /dev/null
> +++ b/Documentation/driver-model/device_link.rst
> @@ -0,0 +1,95 @@
> +============
> +Device links
> +============
> +
> +By default, the driver core only enforces dependencies between devices
> +that are borne out of a parent/child relationship within the device
> +hierarchy: When suspending, resuming or shutting down the system, devices
> +are ordered based on this relationship, i.e. children are always suspended
> +before their parent, and the parent is always resumed before its children.
> +
> +Sometimes there is a need to represent device dependencies beyond the
> +mere parent/child relationship, e.g. between siblings, and have the
> +driver core automatically take care of them.
> +
> +Secondly, the driver core by default does not enforce any driver presence
> +dependencies, i.e. that one device must be bound to a driver before
> +another one can probe or function correctly.
> +
> +Often these two dependency types come together, so a device depends on
> +another one both with regards to driver presence *and* with regards to
> +suspend/resume and shutdown ordering.
> +
> +Device links allow representation of such dependencies in the driver core.
> +
> +In its standard form, a device link combines *both* dependency types:
> +It guarantees correct suspend/resume and shutdown ordering between a
> +"supplier" device and its "consumer" devices, and it guarantees driver
> +presence on the supplier:  The consumer devices are not probed before the
> +supplier is bound to a driver, and they're unbound before the supplier
> +is unbound.
> +
> +When driver presence on the supplier is irrelevant and only correct
> +suspend/resume and shutdown ordering is needed, the device link may
> +simply be set up with the DEVICE_LINK_STATELESS flag.
> +
> +A driver presence dependency between parent and child, i.e. within the
> +regular device hierarchy, could in principle also be represented in the
> +driver core using a device link.
> +
> +If a device link is set up with the DEVICE_LINK_AUTOREMOVE flag, it is
> +automatically purged when the consumer fails to probe or later unbinds.
> +This is handy when adding a device link from the consumer's ->probe hook,
> +as it obviates the need to delete the link in the ->remove hook or in
> +the error path of the ->probe hook.
> +
> +
> +State machine
> +=============
> +
> +"""
> +                .=============================.
> +                |                             |
> +                v                             |
> +DORMANT <=> AVAILABLE <=> CONSUMER_PROBE => ACTIVE
> +   ^                                          |
> +   |                                          |
> +   '============ SUPPLIER_UNBIND <============'
> +"""
> +
> +* The initial state of a device link is passed in to device_link_add().
> +  If the link is created before any devices are probed, it must be set to
> +  DEVICE_LINK_DORMANT.
> +
> +* When a supplier device is bound to a driver, links to its consumers
> +  progress to DEVICE_LINK_AVAILABLE.
> +  (Call to device_links_driver_bound() from driver_bound().)
> +
> +* Before a consumer device is probed, presence of supplier drivers is
> +  verified by checking that links to suppliers are in DEVICE_LINK_AVAILABLE
> +  state.  The state of the links is updated to DEVICE_LINK_CONSUMER_PROBE.
> +  (Call to device_links_check_suppliers() from driver_probe_device().)
> +  This prevents the supplier from unbinding.
> +  (Call to wait_for_device_probe() in device_links_unbind_consumers().)
> +
> +* If the probe fails, links to suppliers revert back to DEVICE_LINK_AVAILABLE.
> +  (Call to device_links_no_driver() from really_probe().)
> +
> +* If the probe succeeds, links to suppliers progress to DEVICE_LINK_ACTIVE.
> +  (Call of device_links_driver_bound() from driver_bound().)
> +
> +* When the consumer's driver is later on removed, links to suppliers revert
> +  back to DEVICE_LINK_AVAILABLE.
> +  (Call to device_links_no_driver() from __device_release_driver().)
> +
> +* Before a supplier's driver is removed, links to consumers that are not
> +  bound to a driver are updated to DEVICE_LINK_SUPPLIER_UNBIND.
> +  (Call to device_links_busy() from __device_release_driver().)
> +  This prevents the consumers from binding.
> +  (Call to device_links_check_suppliers() from driver_probe_device().)
> +  Consumers that are bound are freed from their driver; consumers that are
> +  probing are waited for until they are done.
> +  (Call to device_links_unbind_consumers() from __device_release_driver().)
> +  Once all links to consumers are in DEVICE_LINK_SUPPLIER_UNBIND state,
> +  the supplier driver is released and the links revert to DEVICE_LINK_DORMANT.
> +  (Call to device_links_driver_gone() from __device_release_driver().)

Re: [RFC/RFT][PATCH v3 0/5] Functional dependencies between devices

[Lukas Wunner]

On Wed, Sep 28, 2016 at 02:33:21AM +0200, Rafael J. Wysocki wrote:
> On Tuesday, September 27, 2016 02:34:29 PM Lukas Wunner wrote:
> > I made some notes while reviewing the state machine in patch 2 of this
> > series and thought, why not rework it into something that could eventually
> > go into the Documentation/ tree?
> > 
> > So here's an initial draft.  There's some introductory text plus
> > a description of the state machine.  Just putting this out there now
> > to ease reviewers' lives, despite the obvious WIP status.  I'll try to
> > amend it as the series converges.
> > 
> > This is already rst-formatted but I haven't actually run it through
> > sphinx yet.
> 
> Thanks a lot for doing this!
> 
> It looks good to me in general.  I think it would be good to add it to the
> series at one point (if you don't mind).

Sure thing, thanks.


> I'm only a bit reluctant about advertising the usage of links between
> children and parents, because that doesn't look like the right tool for
> the purpose (as I said before, I'd prefer to add a device flag causing
> the parent driver to be probed before the child one if needed).

That wouldn't cover the unbinding of the child when the parent unbinds
though, so it would only be a subset of the functionality offered by
device links.

I actually don't know of a use case where driver presence is needed
between parent and child.  But the patches look like they should work
out of the box in such a scenario, so I was thinking, why forbid it?
Someone might just try that because they think it should obviously work,
and then they'll find out at runtime that it's forbidden.  That gives
us only a score of 5 in Rusty's API rating scheme.

However for consistency, if you do want to forbid it, I think it should
be forbidden for all ancestors of the device, not just the parent as v3
does it.  (Suspend/resume + shutdown ordering is already handled for
hierarchical dependencies, i.e. all ancestors.)

Thanks,

Lukas

Re: [RFC/RFT][PATCH v3 0/5] Functional dependencies between devices

[Rafael J. Wysocki]

On Wednesday, September 28, 2016 01:42:20 PM Lukas Wunner wrote:
> On Wed, Sep 28, 2016 at 02:33:21AM +0200, Rafael J. Wysocki wrote:
> > On Tuesday, September 27, 2016 02:34:29 PM Lukas Wunner wrote:
> > > I made some notes while reviewing the state machine in patch 2 of this
> > > series and thought, why not rework it into something that could eventually
> > > go into the Documentation/ tree?
> > > 
> > > So here's an initial draft.  There's some introductory text plus
> > > a description of the state machine.  Just putting this out there now
> > > to ease reviewers' lives, despite the obvious WIP status.  I'll try to
> > > amend it as the series converges.
> > > 
> > > This is already rst-formatted but I haven't actually run it through
> > > sphinx yet.
> > 
> > Thanks a lot for doing this!
> > 
> > It looks good to me in general.  I think it would be good to add it to the
> > series at one point (if you don't mind).
> 
> Sure thing, thanks.
> 
> 
> > I'm only a bit reluctant about advertising the usage of links between
> > children and parents, because that doesn't look like the right tool for
> > the purpose (as I said before, I'd prefer to add a device flag causing
> > the parent driver to be probed before the child one if needed).
> 
> That wouldn't cover the unbinding of the child when the parent unbinds
> though, so it would only be a subset of the functionality offered by
> device links.
> 
> I actually don't know of a use case where driver presence is needed
> between parent and child.  But the patches look like they should work
> out of the box in such a scenario, so I was thinking, why forbid it?
> Someone might just try that because they think it should obviously work,
> and then they'll find out at runtime that it's forbidden.  That gives
> us only a score of 5 in Rusty's API rating scheme.
> 
> However for consistency, if you do want to forbid it, I think it should
> be forbidden for all ancestors of the device, not just the parent as v3
> does it.  (Suspend/resume + shutdown ordering is already handled for
> hierarchical dependencies, i.e. all ancestors.)

Well, there is a difference between allowing something to be done and
documenting it as a good idea. :-)

Thanks,
Rafael

Re: [RFC/RFT][PATCH v3 0/5] Functional dependencies between devices

[Lukas Wunner]

On Thu, Sep 29, 2016 at 02:51:45AM +0200, Rafael J. Wysocki wrote:
> On Wednesday, September 28, 2016 01:42:20 PM Lukas Wunner wrote:
> > On Wed, Sep 28, 2016 at 02:33:21AM +0200, Rafael J. Wysocki wrote:
> > > I'm only a bit reluctant about advertising the usage of links between
> > > children and parents, because that doesn't look like the right tool for
> > > the purpose (as I said before, I'd prefer to add a device flag causing
> > > the parent driver to be probed before the child one if needed).
> > 
> > That wouldn't cover the unbinding of the child when the parent unbinds
> > though, so it would only be a subset of the functionality offered by
> > device links.
> > 
> > I actually don't know of a use case where driver presence is needed
> > between parent and child.  But the patches look like they should work
> > out of the box in such a scenario, so I was thinking, why forbid it?
> > Someone might just try that because they think it should obviously work,
> > and then they'll find out at runtime that it's forbidden.  That gives
> > us only a score of 5 in Rusty's API rating scheme.
> > 
> > However for consistency, if you do want to forbid it, I think it should
> > be forbidden for all ancestors of the device, not just the parent as v3
> > does it.  (Suspend/resume + shutdown ordering is already handled for
> > hierarchical dependencies, i.e. all ancestors.)
> 
> Well, there is a difference between allowing something to be done and
> documenting it as a good idea. :-)

I'm reworking the documentation and to address your concerns I have
now reformulated this paragraph as follows:

    To prevent introduction of dependency loops into the graph, it is
    verified upon device link addition that the supplier is not dependent
    on the consumer or any children or consumers of the consumer.
    (Call to device_is_dependent() from device_link_add().)  If that
    constraint is violated, device_link_add() will return %NULL and
    a WARNING will be logged.

    Notably this also prevents addition of a device link from a parent
    device to a child.  However the converse is allowed, i.e. a device link
    from a child to a parent.  Since the driver core already guarantees
    correct suspend/resume and shutdown ordering between parent and child,
    such a device link only makes sense if a driver presence dependency is
    needed on top of that.  In that case driver authors should weigh
    carefully if a device link is the right tool for the purpose.
    A more suitable approach might be to simply use deferred probing or
    add a device flag causing the parent driver to be probed before the
    child one.

If you'd prefer a different wording just shout.

Thanks,

Lukas

[PATCH v4 0/5] Functional dependencies between devices

[Rafael J. Wysocki]

Hi Everyone,

On Thursday, September 08, 2016 11:25:44 PM Rafael J. Wysocki wrote:
> > Hi Everyone,
> > 
> > This is a refresh of the functional dependencies series that I posted last
> > year and which has picked up by Marek quite recently.  For reference,
> > appended is my introductory message sent previously (which may be
> > slightly outdated now).
> > 
> > As last time, the first patch rearranges the code around
> > __device_release_driver() a bit to prepare it for the next one (it
> > actually hasn't changed AFAICS).
> > 
> > The second patch introduces the actual device links mechanics, but without
> > system suspend/resume and runtime PM support which are added by the
> > subsequent patches.
> > 
> > Some bugs found by Marek during his work on these patches should be fixed
> > here.  In particular, the endless recursion in device_reorder_to_tail()
> > which simply was broken before.
> > 
> > There are two additional patches to address the issue with runtime PM
> > support
> > that occured when runtime PM was disabled for some suppliers due to a PM
> > sleep transition in progress.  Those patches simply make runtime PM
> > helpers
> > return 0 in that case which may be controversial, so please let me know if
> > there are concerns about those.
> > 
> > The way device_link_add() works is a bit different, as it takes an
> > additional
> > status argument now.  That makes it possible to create a link in any
> > state,
> > with extra care of course, and should address the problem pointed to by
> > Lukas
> > during the previous discussion.
> > 
> > Also some comments from Tomeu have been addressed.
> 
> An update here.
> 
> The first patch hasn't changed, so I'm resending it.
> 
> The majority of changes in the other patches are in order to address Lukas'
> comments.
> 
> First off, I added a DEVICE_LINK_STATELESS flag that will prevent the driver
> core from trying to maintain device links having it set.
> 
> Also, the DEVICE_LINK_PERSISTENT flag was dropped (as link "persistence" is
> the default behavior now) and there's a new one, DEVICE_LINK_AUTOREMOVE,
> that will cause the driver core to remove the link on the consumer driver
> unbind.
> 
> Moreover, the code checks attempts to create a link between a parent and a
> child device now and actively prevents that from happening.
> 
> The changelog of the second patch has been updated as requested by Ulf.
> 
> The third patch was updated to fix a bug related to the (previously missing)
> clearing of power.direct_complete for supplier devices having consumers that
> don't use direct_complete.
> 
> The next two (runtime PM) patches turned out to be unnecessary, so I've
> dropped them.
> 
> The runtime PM patch [4/5] was reorganized somewhat to reduce the
> indentation
> level in there, but the code flow introduced by it is essentially the same
> and the last patch was simply rebased on top of the new series.

Time for another update. :-)

Fewer changes this time, mostly to address issues found by Lukas and Marek.

The most significant one is to make device_link_add() cope with the case when
the consumer device has not been registered yet when it is called.  The
supplier device still is required to be registered and the function will return
NULL if that is not the case.

Another significant change is in patch [4/5] that now makes the core apply
pm_runtime_get_sync()/pm_runtime_put() to supplier devices around the probing
of a consumer one (in analogy with the parent).

Thanks,
Rafael

[Resend][PATCH v4 1/5] driver core: Add a wrapper around __device_release_driver()

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Add an internal wrapper around __device_release_driver() that will
acquire device locks and do the necessary checks before calling it.

The next patch will make use of it.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/base/dd.c | 30 ++++++++++++++++++------------
 1 file changed, 18 insertions(+), 12 deletions(-)

diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index 16688f50729c..d9e76e9205c7 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -796,6 +796,22 @@ static void __device_release_driver(struct device *dev)
 	}
 }
 
+static void device_release_driver_internal(struct device *dev,
+					   struct device_driver *drv,
+					   struct device *parent)
+{
+	if (parent)
+		device_lock(parent);
+
+	device_lock(dev);
+	if (!drv || drv == dev->driver)
+		__device_release_driver(dev);
+
+	device_unlock(dev);
+	if (parent)
+		device_unlock(parent);
+}
+
 /**
  * device_release_driver - manually detach device from driver.
  * @dev: device.
@@ -810,9 +826,7 @@ void device_release_driver(struct device *dev)
 	 * within their ->remove callback for the same device, they
 	 * will deadlock right here.
 	 */
-	device_lock(dev);
-	__device_release_driver(dev);
-	device_unlock(dev);
+	device_release_driver_internal(dev, NULL, NULL);
 }
 EXPORT_SYMBOL_GPL(device_release_driver);
 
@@ -837,15 +851,7 @@ void driver_detach(struct device_driver *drv)
 		dev = dev_prv->device;
 		get_device(dev);
 		spin_unlock(&drv->p->klist_devices.k_lock);
-
-		if (dev->parent)	/* Needed for USB */
-			device_lock(dev->parent);
-		device_lock(dev);
-		if (dev->driver == drv)
-			__device_release_driver(dev);
-		device_unlock(dev);
-		if (dev->parent)
-			device_unlock(dev->parent);
+		device_release_driver_internal(dev, drv, dev->parent);
 		put_device(dev);
 	}
 }

[PATCH v4 2/5] driver core: Functional dependencies tracking support

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Currently, there is a problem with taking functional dependencies
between devices into account.

What I mean by a "functional dependency" is when the driver of device
B needs device A to be functional and (generally) its driver to be
present in order to work properly.  This has certain consequences
for power management (suspend/resume and runtime PM ordering) and
shutdown ordering of these devices.  In general, it also implies that
the driver of A needs to be working for B to be probed successfully
and it cannot be unbound from the device before the B's driver.

Support for representing those functional dependencies between
devices is added here to allow the driver core to track them and act
on them in certain cases where applicable.

The argument for doing that in the driver core is that there are
quite a few distinct use cases involving device dependencies, they
are relatively hard to get right in a driver (if one wants to
address all of them properly) and it only gets worse if multiplied
by the number of drivers potentially needing to do it.  Morever, at
least one case (asynchronous system suspend/resume) cannot be handled
in a single driver at all, because it requires the driver of A to
wait for B to suspend (during system suspend) and the driver of B to
wait for A to resume (during system resume).

For this reason, represent dependencies between devices as "links",
with the help of struct device_link objects each containing pointers
to the "linked" devices, a list node for each of them, status
information, flags, a lock and an RCU head for synchronization.

Also add two new list heads, links_to_consumers and links_to_suppliers,
to struct device to represent the lists of links to the devices that
depend on the given one (consumers) and to the devices depended on
by it (suppliers), respectively.

The entire data structure consisting of all of the lists of link
objects for all devices is protected by SRCU (for list walking)
and by a mutex (for link object addition/removal).  In addition
to that, each link object has an internal status field whose
value reflects whether or not drivers are bound to the devices
pointed to by the link or probing/removal of their drivers is in
progress etc.  That status field is protected by an internal
spinlock.

New links are added by calling device_link_add() which takes four
arguments: pointers to the devices in question, the initial status
of the link and flags.  In particular, if DEVICE_LINK_STATELESS is
set in the flags, the link status is not to be taken into account
for this link and the driver core will not manage it.  In turn, if
DEVICE_LINK_AUTOREMOVE is set in the flags, the driver core will
remove the link automatically when the consumer device driver
unbinds from it.

One of the actions carried out by device_link_add() is to reorder
the lists used for device shutdown and system suspend/resume to
put the consumer device along with all of its children and all of
its consumers (and so on, recursively) to the ends of those lists
in order to ensure the right ordering between all of the supplier
and consumer devices.

For this reason, it is not possible to create a link between two
devices if the would-be supplier device already depends on the
would-be consumer device as either a direct descendant of it or a
consumer of one of its direct descendants or one of its consumers
and so on.

It also is impossible to create a link between a parent and a child
device (in any direction).

There are two types of link objects, persistent and non-persistent.
The persistent ones stay around until one of the target devices is
deleted, while the non-persistent ones are removed automatically when
the consumer driver unbinds from its device (ie. they are assumed to
be valid only as long as the consumer device has a driver bound to
it).  Persistent links are created by default and non-persistent
links are created when the DEVICE_LINK_AUTOREMOVE flag is passed
to device_link_add().

Both persistent and non-persistent device links can be deleted
explicitly with the help of device_link_del().

Links created without the DEVICE_LINK_STATELESS flag set are managed
by the driver core using a simple state machine.  There are 5 states
each link can be in: DORMANT (unused), AVAILABLE (the supplier driver
is present and functional), CONSUMER_PROBE (the consumer driver is
probing), ACTIVE (both supplier and consumer drivers are present and
functional), and SUPPLIER_UNBIND (the supplier driver is unbinding).
The driver core updates the link state automatically depending on
what happens to the linked devices and for each link state specific
actions are taken in addition to that.

For example, if the supplier driver unbinds from its device, the
driver core will also unbind the drivers of all of its consumers
automatically under the assumption that they cannot function
properly without the supplier.  Analogously, the driver core will
only allow the consumer driver to bind to its device if the
supplier driver is present and functional (ie. the link is in
the AVAILABLE state).  If that's not the case, it will rely on
the existing deferred probing mechanism to wait for the supplier
driver to become available.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---

v3 -> v4:

- Add the in_dpm_list field to struct dev_pm_info and use it for checking
  if the device has been added to dpm_list already, which is needed for
  handling the "not registered consumer" case in device_link_add().

- Rework device_link_add() to handle consumer devices that have not been
  registered before calling it.

- Drop the parent check from device_link_add().

- Rename device_links_driver_gone() to device_links_driver_cleanup()

- Rearrange device_links_unbind_consumers() to avoid one spin_unlock()
  instance (which wasn't necessary).

- Introduce device_links_purge() to delete links from a device going away
  and make device_del() call it (instead of running links-related code
  directly).

- Move the invocation of device_links_ckeck_suppliers() to really_probe().

- Change the description of the DEVICE_LINK_STATELESS flag.

---
 drivers/base/base.h        |   11 +
 drivers/base/core.c        |  491 +++++++++++++++++++++++++++++++++++++++++++++
 drivers/base/dd.c          |   42 +++
 drivers/base/power/main.c  |    2 
 drivers/base/power/power.h |   10 
 include/linux/device.h     |   38 +++
 include/linux/pm.h         |    1 
 7 files changed, 590 insertions(+), 5 deletions(-)

Index: linux-pm/drivers/base/base.h
===================================================================
--- linux-pm.orig/drivers/base/base.h
+++ linux-pm/drivers/base/base.h
@@ -107,6 +107,9 @@ extern void bus_remove_device(struct dev
 
 extern int bus_add_driver(struct device_driver *drv);
 extern void bus_remove_driver(struct device_driver *drv);
+extern void device_release_driver_internal(struct device *dev,
+					   struct device_driver *drv,
+					   struct device *parent);
 
 extern void driver_detach(struct device_driver *drv);
 extern int driver_probe_device(struct device_driver *drv, struct device *dev);
@@ -152,3 +155,11 @@ extern int devtmpfs_init(void);
 #else
 static inline int devtmpfs_init(void) { return 0; }
 #endif
+
+/* Device links support */
+extern int device_links_check_suppliers(struct device *dev);
+extern void device_links_driver_bound(struct device *dev);
+extern void device_links_driver_cleanup(struct device *dev);
+extern void device_links_no_driver(struct device *dev);
+extern bool device_links_busy(struct device *dev);
+extern void device_links_unbind_consumers(struct device *dev);
Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -44,6 +44,493 @@ static int __init sysfs_deprecated_setup
 early_param("sysfs.deprecated", sysfs_deprecated_setup);
 #endif
 
+/* Device links support. */
+
+DEFINE_STATIC_SRCU(device_links_srcu);
+static DEFINE_MUTEX(device_links_lock);
+
+/**
+ * device_is_dependent - Check if one device depends on another one
+ * @dev: Device to check dependencies for.
+ * @target: Device to check against.
+ *
+ * Check if @target depends on @dev or any device dependent on it (its child or
+ * its consumer etc).  Return 1 if that is the case or 0 otherwise.
+ */
+static int device_is_dependent(struct device *dev, void *target)
+{
+	struct device_link *link;
+	int ret;
+
+	if (WARN_ON(dev == target))
+		return 1;
+
+	ret = device_for_each_child(dev, target, device_is_dependent);
+	if (ret)
+		return ret;
+
+	list_for_each_entry(link, &dev->links_to_consumers, s_node) {
+		if (WARN_ON(link->consumer == target))
+			return 1;
+
+		ret = device_is_dependent(link->consumer, target);
+		if (ret)
+			break;
+	}
+	return ret;
+}
+
+static int device_reorder_to_tail(struct device *dev, void *not_used)
+{
+	struct device_link *link;
+
+	/*
+	 * Devices that have not been registered yet will be put to the ends
+	 * of the lists during the registratio, so skip them here.
+	 */
+	if (device_is_registered(dev))
+		devices_kset_move_last(dev);
+
+	if (device_pm_initialized(dev))
+		device_pm_move_last(dev);
+
+	device_for_each_child(dev, NULL, device_reorder_to_tail);
+	list_for_each_entry(link, &dev->links_to_consumers, s_node)
+		device_reorder_to_tail(link->consumer, NULL);
+
+	return 0;
+}
+
+/**
+ * device_link_add - Create a link between two devices.
+ * @consumer: Consumer end of the link.
+ * @supplier: Supplier end of the link.
+ * @status: The initial status of the link.
+ * @flags: Link flags.
+ *
+ * If the DEVICE_LINK_STATELESS flag is set, @status is ignored.  Otherwise,
+ * the caller is responsible for ensuring that @status reflects the current
+ * status of both @consumer and @supplier.
+ *
+ * If the DEVICE_LINK_AUTOREMOVE is set, the link will be removed automatically
+ * when the consumer device driver unbinds from it.  The combination of both
+ * DEVICE_LINK_AUTOREMOVE and DEVICE_LINK_STATELESS set is invalid and will
+ * cause NULL to be returned.
+ *
+ * A side effect of the link creation is re-ordering of dpm_list and the
+ * devices_kset list by moving the consumer device and all devices depending
+ * on it to the ends of these lists (that does not happen to devices that have
+ * not been registered when this function is called).
+ *
+ * The supplier device is required to be registered when this function is called
+ * and NULL will be returned if that is not the case.  The consumer device need
+ * not be registerd, however.
+ */
+struct device_link *device_link_add(struct device *consumer,
+				    struct device *supplier,
+				    enum device_link_status status, u32 flags)
+{
+	struct device_link *link;
+
+	if (!consumer || !supplier ||
+	    ((flags & DEVICE_LINK_STATELESS) && (flags & DEVICE_LINK_AUTOREMOVE)))
+		return NULL;
+
+	mutex_lock(&device_links_lock);
+	device_pm_lock();
+
+	/*
+	 * If the supplier has not been fully registered yet or there is a
+	 * reverse dependency between the consumer and the supplier already in
+	 * the graph, return NULL.
+	 */
+	if (!device_pm_initialized(supplier)
+	    || device_is_dependent(consumer, supplier)) {
+		link = NULL;
+		goto out;
+	}
+
+	list_for_each_entry(link, &supplier->links_to_consumers, s_node)
+		if (link->consumer == consumer)
+			goto out;
+
+	link = kmalloc(sizeof(*link), GFP_KERNEL);
+	if (!link)
+		goto out;
+
+	get_device(supplier);
+	link->supplier = supplier;
+	INIT_LIST_HEAD(&link->s_node);
+	get_device(consumer);
+	link->consumer = consumer;
+	INIT_LIST_HEAD(&link->c_node);
+	spin_lock_init(&link->lock);
+	link->flags = flags;
+	link->status = (flags & DEVICE_LINK_STATELESS) ?
+					DEVICE_LINK_NO_STATE : status;
+
+	/*
+	 * Move the consumer and all of the devices depending on it to the end
+	 * of dpm_list and the devices_kset list.
+	 *
+	 * It is necessary to hold dpm_list locked throughout all that or else
+	 * we may end up suspending with a wrong ordering of it.
+	 */
+	device_reorder_to_tail(consumer, NULL);
+
+	list_add_tail_rcu(&link->s_node, &supplier->links_to_consumers);
+	list_add_tail_rcu(&link->c_node, &consumer->links_to_suppliers);
+
+	dev_info(consumer, "Linked as a consumer to %s\n", dev_name(supplier));
+
+ out:
+	device_pm_unlock();
+	mutex_unlock(&device_links_lock);
+	return link;
+}
+EXPORT_SYMBOL_GPL(device_link_add);
+
+static void __device_link_free_srcu(struct rcu_head *rhead)
+{
+	struct device_link *link;
+
+	link = container_of(rhead, struct device_link, rcu_head);
+	put_device(link->consumer);
+	put_device(link->supplier);
+	kfree(link);
+}
+
+static void __device_link_del(struct device_link *link)
+{
+	dev_info(link->consumer, "Dropping the link to %s\n",
+		 dev_name(link->supplier));
+
+	list_del_rcu(&link->s_node);
+	list_del_rcu(&link->c_node);
+	call_srcu(&device_links_srcu, &link->rcu_head, __device_link_free_srcu);
+}
+
+/**
+ * device_link_del - Delete a link between two devices.
+ * @link: Device link to delete.
+ *
+ * The caller must ensure proper synchronization of this function with runtime
+ * PM.
+ */
+void device_link_del(struct device_link *link)
+{
+	mutex_lock(&device_links_lock);
+	device_pm_lock();
+	__device_link_del(link);
+	device_pm_unlock();
+	mutex_unlock(&device_links_lock);
+}
+EXPORT_SYMBOL_GPL(device_link_del);
+
+static int device_links_read_lock(void)
+{
+	return srcu_read_lock(&device_links_srcu);
+}
+
+static void device_links_read_unlock(int idx)
+{
+	return srcu_read_unlock(&device_links_srcu, idx);
+}
+
+static void device_links_missing_supplier(struct device *dev)
+{
+	struct device_link *link;
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
+		spin_lock(&link->lock);
+
+		if (link->status == DEVICE_LINK_CONSUMER_PROBE)
+			link->status = DEVICE_LINK_AVAILABLE;
+
+		spin_unlock(&link->lock);
+	}
+}
+
+/**
+ * device_links_check_suppliers - Check presence of supplier drivers.
+ * @dev: Consumer device.
+ *
+ * Check links from this device to any suppliers.  Walk the list of the device's
+ * consumer links and see if all of the suppliers are available.  If not, simply
+ * return -EPROBE_DEFER.
+ *
+ * Walk the list under SRCU and check each link's status field under its lock.
+ *
+ * We need to guarantee that the supplier will not go away after the check has
+ * been positive here.  It only can go away in __device_release_driver() and
+ * that function  checks the device's links to consumers.  This means we need to
+ * mark the link as "consumer probe in progress" to make the supplier removal
+ * wait for us to complete (or bad things may happen).
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ */
+int device_links_check_suppliers(struct device *dev)
+{
+	struct device_link *link;
+	int idx, ret = 0;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		spin_lock(&link->lock);
+		if (link->status != DEVICE_LINK_AVAILABLE) {
+			spin_unlock(&link->lock);
+			device_links_missing_supplier(dev);
+			ret = -EPROBE_DEFER;
+			break;
+		}
+		link->status = DEVICE_LINK_CONSUMER_PROBE;
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+	return ret;
+}
+
+/**
+ * device_links_driver_bound - Update device links after probing its driver.
+ * @dev: Device to update the links for.
+ *
+ * The probe has been successful, so update links from this device to any
+ * consumers by changing their status to "available".
+ *
+ * Also change the status of @dev's links to suppliers to "active".
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ */
+void device_links_driver_bound(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		spin_lock(&link->lock);
+		WARN_ON(link->status != DEVICE_LINK_DORMANT);
+		link->status = DEVICE_LINK_AVAILABLE;
+		spin_unlock(&link->lock);
+	}
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		spin_lock(&link->lock);
+		WARN_ON(link->status != DEVICE_LINK_CONSUMER_PROBE);
+		link->status = DEVICE_LINK_ACTIVE;
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+}
+
+/**
+ * device_links_driver_cleanup - Update links after driver removal.
+ * @dev: Device whose driver has just gone away.
+ *
+ * Update links to consumers for @dev by changing their status to "dormant".
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ */
+void device_links_driver_cleanup(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		WARN_ON(link->flags & DEVICE_LINK_AUTOREMOVE);
+		spin_lock(&link->lock);
+		WARN_ON(link->status != DEVICE_LINK_SUPPLIER_UNBIND);
+		link->status = DEVICE_LINK_DORMANT;
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+}
+
+/**
+ * device_links_no_driver - Update links of a device without a driver.
+ * @dev: Device without a drvier.
+ *
+ * Delete all non-persistent links from this device to any suppliers.
+ *
+ * Persistent links stay around, but their status is changed to "available",
+ * unless they already are in the "supplier unbind in progress" state in which
+ * case they need not be updated.
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ */
+void device_links_no_driver(struct device *dev)
+{
+	struct device_link *link, *ln;
+
+	mutex_lock(&device_links_lock);
+
+	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_suppliers, c_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		if (link->flags & DEVICE_LINK_AUTOREMOVE) {
+			__device_link_del(link);
+		} else {
+			spin_lock(&link->lock);
+
+			if (link->status != DEVICE_LINK_SUPPLIER_UNBIND)
+				link->status = DEVICE_LINK_AVAILABLE;
+
+			spin_unlock(&link->lock);
+		}
+	}
+
+	mutex_unlock(&device_links_lock);
+}
+
+/**
+ * device_links_busy - Check if there are any busy links to consumers.
+ * @dev: Device to check.
+ *
+ * Check each consumer of the device and return 'true' if its link's status
+ * is one of "consumer probe" or "active" (meaning that the given consumer is
+ * probing right now or its driver is present).  Otherwise, change the link
+ * state to "supplier unbind" to prevent the consumer from being probed
+ * successfully going forward.
+ *
+ * Return 'false' if there are no probing or active consumers.
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ */
+bool device_links_busy(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+	bool ret = false;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		spin_lock(&link->lock);
+		if (link->status == DEVICE_LINK_CONSUMER_PROBE
+		    || link->status == DEVICE_LINK_ACTIVE) {
+			spin_unlock(&link->lock);
+			ret = true;
+			break;
+		}
+		link->status = DEVICE_LINK_SUPPLIER_UNBIND;
+		spin_unlock(&link->lock);
+	}
+
+	device_links_read_unlock(idx);
+	return ret;
+}
+
+/**
+ * device_links_unbind_consumers - Force unbind consumers of the given device.
+ * @dev: Device to unbind the consumers of.
+ *
+ * Walk the list of links to consumers for @dev and if any of them is in the
+ * "consumer probe" state, wait for all device probes in progress to complete
+ * and start over.
+ *
+ * If that's not the case, change the status of the link to "supplier unbind"
+ * and check if the link was in the "active" state.  If so, force the consumer
+ * driver to unbind and start over (the consumer will not re-probe as we have
+ * changed the state of the link already).
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ */
+void device_links_unbind_consumers(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+ start:
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
+		enum device_link_status status;
+
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		spin_lock(&link->lock);
+		status = link->status;
+		if (status == DEVICE_LINK_CONSUMER_PROBE) {
+			spin_unlock(&link->lock);
+
+			device_links_read_unlock(idx);
+
+			wait_for_device_probe();
+			goto start;
+		}
+		link->status = DEVICE_LINK_SUPPLIER_UNBIND;
+		spin_unlock(&link->lock);
+		if (status == DEVICE_LINK_ACTIVE) {
+			struct device *consumer = link->consumer;
+
+			get_device(consumer);
+
+			device_links_read_unlock(idx);
+
+			device_release_driver_internal(consumer, NULL,
+						       consumer->parent);
+			put_device(consumer);
+			goto start;
+		}
+	}
+
+	device_links_read_unlock(idx);
+}
+
+/**
+ * device_links_purge - Delete existing links to other devices.
+ * @dev: Target device.
+ */
+static void device_links_purge(struct device *dev)
+{
+	struct device_link *link, *ln;
+
+	/*
+	 * Delete all of the remaining links from this device to any other
+	 * devices (either consumers or suppliers).
+	 */
+	mutex_lock(&device_links_lock);
+
+	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_suppliers, c_node) {
+		WARN_ON(link->status == DEVICE_LINK_ACTIVE);
+		__device_link_del(link);
+	}
+
+	list_for_each_entry_safe_reverse(link, ln, &dev->links_to_consumers, s_node) {
+		WARN_ON(link->status != DEVICE_LINK_DORMANT &&
+			link->status != DEVICE_LINK_NO_STATE);
+		__device_link_del(link);
+	}
+
+	mutex_unlock(&device_links_lock);
+}
+
+/* Device links support end. */
+
 int (*platform_notify)(struct device *dev) = NULL;
 int (*platform_notify_remove)(struct device *dev) = NULL;
 static struct kobject *dev_kobj;
@@ -711,6 +1198,8 @@ void device_initialize(struct device *de
 #ifdef CONFIG_GENERIC_MSI_IRQ
 	INIT_LIST_HEAD(&dev->msi_list);
 #endif
+	INIT_LIST_HEAD(&dev->links_to_consumers);
+	INIT_LIST_HEAD(&dev->links_to_suppliers);
 }
 EXPORT_SYMBOL_GPL(device_initialize);
 
@@ -1240,6 +1729,8 @@ void device_del(struct device *dev)
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DEL_DEVICE, dev);
+
+	device_links_purge(dev);
 	dpm_sysfs_remove(dev);
 	if (parent)
 		klist_del(&dev->p->knode_parent);
Index: linux-pm/drivers/base/dd.c
===================================================================
--- linux-pm.orig/drivers/base/dd.c
+++ linux-pm/drivers/base/dd.c
@@ -249,6 +249,7 @@ static void driver_bound(struct device *
 		 __func__, dev_name(dev));
 
 	klist_add_tail(&dev->p->knode_driver, &dev->driver->p->klist_devices);
+	device_links_driver_bound(dev);
 
 	device_pm_check_callbacks(dev);
 
@@ -341,6 +342,10 @@ static int really_probe(struct device *d
 		return ret;
 	}
 
+	ret = device_links_check_suppliers(dev);
+	if (ret)
+		return ret;
+
 	atomic_inc(&probe_count);
 	pr_debug("bus: '%s': %s: probing driver %s with device %s\n",
 		 drv->bus->name, __func__, drv->name, dev_name(dev));
@@ -399,6 +404,7 @@ probe_failed:
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
 pinctrl_bind_failed:
+	device_links_no_driver(dev);
 	devres_release_all(dev);
 	driver_sysfs_remove(dev);
 	dev->driver = NULL;
@@ -756,7 +762,7 @@ EXPORT_SYMBOL_GPL(driver_attach);
  * __device_release_driver() must be called with @dev lock held.
  * When called for a USB interface, @dev->parent lock must be held as well.
  */
-static void __device_release_driver(struct device *dev)
+static void __device_release_driver(struct device *dev, struct device *parent)
 {
 	struct device_driver *drv;
 
@@ -765,6 +771,25 @@ static void __device_release_driver(stru
 		if (driver_allows_async_probing(drv))
 			async_synchronize_full();
 
+		while (device_links_busy(dev)) {
+			device_unlock(dev);
+			if (parent)
+				device_unlock(parent);
+
+			device_links_unbind_consumers(dev);
+			if (parent)
+				device_lock(parent);
+
+			device_lock(dev);
+			/*
+			 * A concurrent invocation of the same function might
+			 * have released the driver successfully while this one
+			 * was waiting, so check for that.
+			 */
+			if (dev->driver != drv)
+				return;
+		}
+
 		pm_runtime_get_sync(dev);
 
 		driver_sysfs_remove(dev);
@@ -780,6 +805,9 @@ static void __device_release_driver(stru
 			dev->bus->remove(dev);
 		else if (drv->remove)
 			drv->remove(dev);
+
+		device_links_driver_cleanup(dev);
+		device_links_no_driver(dev);
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);
@@ -796,16 +824,16 @@ static void __device_release_driver(stru
 	}
 }
 
-static void device_release_driver_internal(struct device *dev,
-					   struct device_driver *drv,
-					   struct device *parent)
+void device_release_driver_internal(struct device *dev,
+				    struct device_driver *drv,
+				    struct device *parent)
 {
 	if (parent)
 		device_lock(parent);
 
 	device_lock(dev);
 	if (!drv || drv == dev->driver)
-		__device_release_driver(dev);
+		__device_release_driver(dev, parent);
 
 	device_unlock(dev);
 	if (parent)
@@ -818,6 +846,10 @@ static void device_release_driver_intern
  *
  * Manually detach device from driver.
  * When called for a USB interface, @dev->parent lock must be held.
+ *
+ * If this function is to be called with @dev->parent lock held, ensure that
+ * the device's consumers are unbound in advance or that their locks can be
+ * acquired under the @dev->parent lock.
  */
 void device_release_driver(struct device *dev)
 {
Index: linux-pm/include/linux/device.h
===================================================================
--- linux-pm.orig/include/linux/device.h
+++ linux-pm/include/linux/device.h
@@ -706,6 +706,35 @@ struct device_dma_parameters {
 	unsigned long segment_boundary_mask;
 };
 
+enum device_link_status {
+	DEVICE_LINK_NO_STATE = -1,
+	DEVICE_LINK_DORMANT = 0,	/* Link not in use. */
+	DEVICE_LINK_AVAILABLE,		/* Supplier driver is present. */
+	DEVICE_LINK_ACTIVE,		/* Consumer driver is present too. */
+	DEVICE_LINK_CONSUMER_PROBE,	/* Consumer is probing. */
+	DEVICE_LINK_SUPPLIER_UNBIND,	/* Supplier is unbinding. */
+};
+
+/*
+ * Device link flags.
+ *
+ * STATELESS: The core won't track the presence of supplier/consumer drivers.
+ * AUTOREMOVE: Remove this link automatically on cunsumer driver unbind.
+ */
+#define DEVICE_LINK_STATELESS	(1 << 0)
+#define DEVICE_LINK_AUTOREMOVE	(1 << 1)
+
+struct device_link {
+	struct device *supplier;
+	struct list_head s_node;
+	struct device *consumer;
+	struct list_head c_node;
+	enum device_link_status status;
+	u32 flags;
+	spinlock_t lock;
+	struct rcu_head rcu_head;
+};
+
 /**
  * struct device - The basic device structure
  * @parent:	The device's "parent" device, the device to which it is attached.
@@ -731,6 +760,8 @@ struct device_dma_parameters {
  * 		on.  This shrinks the "Board Support Packages" (BSPs) and
  * 		minimizes board-specific #ifdefs in drivers.
  * @driver_data: Private pointer for driver specific info.
+ * @links_to_consumers: Links to consumer devices.
+ * @links_to_suppliers: Links to supplier devices.
  * @power:	For device power management.
  * 		See Documentation/power/devices.txt for details.
  * @pm_domain:	Provide callbacks that are executed during system suspend,
@@ -797,6 +828,8 @@ struct device {
 					   core doesn't touch it */
 	void		*driver_data;	/* Driver data, set and get with
 					   dev_set/get_drvdata */
+	struct list_head	links_to_consumers;
+	struct list_head	links_to_suppliers;
 	struct dev_pm_info	power;
 	struct dev_pm_domain	*pm_domain;
 
@@ -1113,6 +1146,11 @@ extern void device_shutdown(void);
 /* debugging and troubleshooting/diagnostic helpers. */
 extern const char *dev_driver_string(const struct device *dev);
 
+/* Device links interface. */
+struct device_link *device_link_add(struct device *consumer,
+				    struct device *supplier,
+				    enum device_link_status status, u32 flags);
+void device_link_del(struct device_link *link);
 
 #ifdef CONFIG_PRINTK
 
Index: linux-pm/drivers/base/power/main.c
===================================================================
--- linux-pm.orig/drivers/base/power/main.c
+++ linux-pm/drivers/base/power/main.c
@@ -131,6 +131,7 @@ void device_pm_add(struct device *dev)
 		dev_warn(dev, "parent %s should not be sleeping\n",
 			dev_name(dev->parent));
 	list_add_tail(&dev->power.entry, &dpm_list);
+	dev->power.in_dpm_list = true;
 	mutex_unlock(&dpm_list_mtx);
 }
 
@@ -145,6 +146,7 @@ void device_pm_remove(struct device *dev
 	complete_all(&dev->power.completion);
 	mutex_lock(&dpm_list_mtx);
 	list_del_init(&dev->power.entry);
+	dev->power.in_dpm_list = false;
 	mutex_unlock(&dpm_list_mtx);
 	device_wakeup_disable(dev);
 	pm_runtime_remove(dev);
Index: linux-pm/drivers/base/power/power.h
===================================================================
--- linux-pm.orig/drivers/base/power/power.h
+++ linux-pm/drivers/base/power/power.h
@@ -127,6 +127,11 @@ extern void device_pm_move_after(struct
 extern void device_pm_move_last(struct device *);
 extern void device_pm_check_callbacks(struct device *dev);
 
+static inline bool device_pm_initialized(struct device *dev)
+{
+	return dev->power.in_dpm_list;
+}
+
 #else /* !CONFIG_PM_SLEEP */
 
 static inline void device_pm_sleep_init(struct device *dev) {}
@@ -146,6 +151,11 @@ static inline void device_pm_move_last(s
 
 static inline void device_pm_check_callbacks(struct device *dev) {}
 
+static inline bool device_pm_initialized(struct device *dev)
+{
+	return device_is_registered(dev);
+}
+
 #endif /* !CONFIG_PM_SLEEP */
 
 static inline void device_pm_init(struct device *dev)
Index: linux-pm/include/linux/pm.h
===================================================================
--- linux-pm.orig/include/linux/pm.h
+++ linux-pm/include/linux/pm.h
@@ -559,6 +559,7 @@ struct dev_pm_info {
 	pm_message_t		power_state;
 	unsigned int		can_wakeup:1;
 	unsigned int		async_suspend:1;
+	bool			in_dpm_list:1;	/* Owned by the PM core */
 	bool			is_prepared:1;	/* Owned by the PM core */
 	bool			is_suspended:1;	/* Ditto */
 	bool			is_noirq_suspended:1;

Re: [PATCH v4 2/5] driver core: Functional dependencies tracking support

[Lukas Wunner]

On Thu, Sep 29, 2016 at 02:38:04AM +0200, Rafael J. Wysocki wrote:
> +static int device_reorder_to_tail(struct device *dev, void *not_used)
> +{
> +	struct device_link *link;
> +
> +	/*
> +	 * Devices that have not been registered yet will be put to the ends
> +	 * of the lists during the registratio, so skip them here.
                                              ^
                                              n

> +     if (device_is_registered(dev))
> +             devices_kset_move_last(dev);
> +
> +     if (device_pm_initialized(dev))
> +             device_pm_move_last(dev);

Clever solution to this problem.  So little code!


> +/**
> + * device_links_check_suppliers - Check presence of supplier drivers.
> + * @dev: Consumer device.
> + *
> + * Check links from this device to any suppliers.  Walk the list of the device's
> + * consumer links and see if all of the suppliers are available.  If not, simply
      ^^^^^^^^
     "supplier links and see if all if them are available."


> +/*
> + * Device link flags.
> + *
> + * STATELESS: The core won't track the presence of supplier/consumer drivers.
> + * AUTOREMOVE: Remove this link automatically on cunsumer driver unbind.
                                                     ^
                                                     o


Apart from these nits patch [2/5] LGTM, so FWIW:

Reviewed-by: Lukas Wunner <lukas@wunner.de>

Thanks,

Lukas

Re: [PATCH v4 2/5] driver core: Functional dependencies tracking support

[Rafael J. Wysocki]

On Sat, Oct 1, 2016 at 9:43 AM, Lukas Wunner <lukas@wunner.de> wrote:
> On Thu, Sep 29, 2016 at 02:38:04AM +0200, Rafael J. Wysocki wrote:
>> +static int device_reorder_to_tail(struct device *dev, void *not_used)
>> +{
>> +     struct device_link *link;
>> +
>> +     /*
>> +      * Devices that have not been registered yet will be put to the ends
>> +      * of the lists during the registratio, so skip them here.
>                                               ^
>                                               n
>
>> +     if (device_is_registered(dev))
>> +             devices_kset_move_last(dev);
>> +
>> +     if (device_pm_initialized(dev))
>> +             device_pm_move_last(dev);
>
> Clever solution to this problem.  So little code!
>
>
>> +/**
>> + * device_links_check_suppliers - Check presence of supplier drivers.
>> + * @dev: Consumer device.
>> + *
>> + * Check links from this device to any suppliers.  Walk the list of the device's
>> + * consumer links and see if all of the suppliers are available.  If not, simply
>       ^^^^^^^^
>      "supplier links and see if all if them are available."
>
>
>> +/*
>> + * Device link flags.
>> + *
>> + * STATELESS: The core won't track the presence of supplier/consumer drivers.
>> + * AUTOREMOVE: Remove this link automatically on cunsumer driver unbind.
>                                                      ^
>                                                      o
>
>
> Apart from these nits patch [2/5] LGTM, so FWIW:
>
> Reviewed-by: Lukas Wunner <lukas@wunner.de>

Thanks for the review, much appreciated!

Cheers,
Rafael

[Resend][PATCH v4 3/5] PM / sleep: Make async suspend/resume of devices use device links

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Make the device suspend/resume part of the core system
suspend/resume code use device links to ensure that supplier
and consumer devices will be suspended and resumed in the right
order in case of async suspend/resume.

The idea, roughly, is to use dpm_wait() to wait for all consumers
before a supplier device suspend and to wait for all suppliers
before a consumer device resume.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/base/base.h       |    2 +
 drivers/base/core.c       |    4 +-
 drivers/base/power/main.c |   85 ++++++++++++++++++++++++++++++++++++++++++----
 3 files changed, 83 insertions(+), 8 deletions(-)

Index: linux-pm/drivers/base/base.h
===================================================================
--- linux-pm.orig/drivers/base/base.h
+++ linux-pm/drivers/base/base.h
@@ -163,3 +163,5 @@ extern void device_links_driver_cleanup(
 extern void device_links_no_driver(struct device *dev);
 extern bool device_links_busy(struct device *dev);
 extern void device_links_unbind_consumers(struct device *dev);
+extern int device_links_read_lock(void);
+extern void device_links_read_unlock(int idx);
Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -227,12 +227,12 @@ void device_link_del(struct device_link
 }
 EXPORT_SYMBOL_GPL(device_link_del);
 
-static int device_links_read_lock(void)
+int device_links_read_lock(void)
 {
 	return srcu_read_lock(&device_links_srcu);
 }
 
-static void device_links_read_unlock(int idx)
+void device_links_read_unlock(int idx)
 {
 	return srcu_read_unlock(&device_links_srcu, idx);
 }
Index: linux-pm/drivers/base/power/main.c
===================================================================
--- linux-pm.orig/drivers/base/power/main.c
+++ linux-pm/drivers/base/power/main.c
@@ -246,6 +246,62 @@ static void dpm_wait_for_children(struct
        device_for_each_child(dev, &async, dpm_wait_fn);
 }
 
+static void dpm_wait_for_suppliers(struct device *dev, bool async)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	/*
+	 * If the supplier goes away right after we've checked the link to it,
+	 * we'll wait for its completion to change the state, but that's fine,
+	 * because the only things that will block as a result are the SRCU
+	 * callbacks freeing the link objects for the links in the list we're
+	 * walking.
+	 */
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node)
+		if (link->status != DEVICE_LINK_DORMANT)
+			dpm_wait(link->supplier, async);
+
+	device_links_read_unlock(idx);
+}
+
+static void dpm_wait_for_superior(struct device *dev, bool async)
+{
+	dpm_wait(dev->parent, async);
+	dpm_wait_for_suppliers(dev, async);
+}
+
+static void dpm_wait_for_consumers(struct device *dev, bool async)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	/*
+	 * The status of a device link can only be changed from "dormant" by a
+	 * probe, but that cannot happen during system suspend/resume.  In
+	 * theory it can change to "dormant" at that time, but then it is
+	 * reasonable to wait for the target device anyway (eg. if it goes
+	 * away, it's better to wait for it to go away completely and then
+	 * continue instead of trying to continue in parallel with its
+	 * unregistration).
+	 */
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node)
+		if (link->status != DEVICE_LINK_DORMANT)
+			dpm_wait(link->consumer, async);
+
+	device_links_read_unlock(idx);
+}
+
+static void dpm_wait_for_subordinate(struct device *dev, bool async)
+{
+	dpm_wait_for_children(dev, async);
+	dpm_wait_for_consumers(dev, async);
+}
+
 /**
  * pm_op - Return the PM operation appropriate for given PM event.
  * @ops: PM operations to choose from.
@@ -490,7 +546,7 @@ static int device_resume_noirq(struct de
 	if (!dev->power.is_noirq_suspended)
 		goto Out;
 
-	dpm_wait(dev->parent, async);
+	dpm_wait_for_superior(dev, async);
 
 	if (dev->pm_domain) {
 		info = "noirq power domain ";
@@ -620,7 +676,7 @@ static int device_resume_early(struct de
 	if (!dev->power.is_late_suspended)
 		goto Out;
 
-	dpm_wait(dev->parent, async);
+	dpm_wait_for_superior(dev, async);
 
 	if (dev->pm_domain) {
 		info = "early power domain ";
@@ -752,7 +808,7 @@ static int device_resume(struct device *
 		goto Complete;
 	}
 
-	dpm_wait(dev->parent, async);
+	dpm_wait_for_superior(dev, async);
 	dpm_watchdog_set(&wd, dev);
 	device_lock(dev);
 
@@ -1040,7 +1096,7 @@ static int __device_suspend_noirq(struct
 	if (dev->power.syscore || dev->power.direct_complete)
 		goto Complete;
 
-	dpm_wait_for_children(dev, async);
+	dpm_wait_for_subordinate(dev, async);
 
 	if (dev->pm_domain) {
 		info = "noirq power domain ";
@@ -1187,7 +1243,7 @@ static int __device_suspend_late(struct
 	if (dev->power.syscore || dev->power.direct_complete)
 		goto Complete;
 
-	dpm_wait_for_children(dev, async);
+	dpm_wait_for_subordinate(dev, async);
 
 	if (dev->pm_domain) {
 		info = "late power domain ";
@@ -1344,6 +1400,22 @@ static int legacy_suspend(struct device
 	return error;
 }
 
+static void dpm_clear_suppliers_direct_complete(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
+		spin_lock_irq(&link->supplier->power.lock);
+		link->supplier->power.direct_complete = false;
+		spin_unlock_irq(&link->supplier->power.lock);
+	}
+
+	device_links_read_unlock(idx);
+}
+
 /**
  * device_suspend - Execute "suspend" callbacks for given device.
  * @dev: Device to handle.
@@ -1360,7 +1432,7 @@ static int __device_suspend(struct devic
 	TRACE_DEVICE(dev);
 	TRACE_SUSPEND(0);
 
-	dpm_wait_for_children(dev, async);
+	dpm_wait_for_subordinate(dev, async);
 
 	if (async_error)
 		goto Complete;
@@ -1456,6 +1528,7 @@ static int __device_suspend(struct devic
 
 			spin_unlock_irq(&parent->power.lock);
 		}
+		dpm_clear_suppliers_direct_complete(dev);
 	}
 
 	device_unlock(dev);

[PATCH v4 4/5] PM / runtime: Use device links

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Modify the runtime PM framework to use device links to ensure that
supplier devices will not be suspended if any of their consumer
devices are active.

The idea is to reference count suppliers on the consumer's resume
and drop references to them on its suspend.  The information on
whether or not the supplier has been reference counted by the
consumer's (runtime) resume is stored in a new field (rpm_active)
in the link object for each link.

It may be necessary to clean up those references when the
supplier is unbinding and that's why the links whose status is
DEVICE_LINK_SUPPLIER_UNBIND are skipped by the runtime suspend
and resume code.

The above means that if the consumer device is probed in the
runtime-active state, the supplier has to be resumed and reference
counted by device_link_add() so the code works as expected on its
(runtime) suspend.  There is a new flag, DEVICE_LINK_RPM_ACTIVE,
to tell device_link_add() about that (in which case the caller
is responsible for making sure that the consumer really will
be runtime-active when runtime PM is enabled for it).

The other new link flag, DEVICE_LINK_PM_RUNTIME, tells the core
whether or not the link should be used for runtime PM at all.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---

v3 -> v4:

- Add pm_runtime_get/put_suppliers() and make the core call them
  around the probing of a consumer device (in analogy with what it
  does to the device's parent).

---
 drivers/base/core.c          |   23 +++++-
 drivers/base/dd.c            |    3 
 drivers/base/power/runtime.c |  157 +++++++++++++++++++++++++++++++++++++++++--
 include/linux/device.h       |    5 +
 include/linux/pm_runtime.h   |    6 +
 5 files changed, 186 insertions(+), 8 deletions(-)

Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -109,8 +109,16 @@ static int device_reorder_to_tail(struct
  * @flags: Link flags.
  *
  * If the DEVICE_LINK_STATELESS flag is set, @status is ignored.  Otherwise,
- * the caller is responsible for ensuring that @status reflects the current
- * status of both @consumer and @supplier.
+ * the caller is responsible for ensuring that (a) @status reflects the current
+ * status of both @consumer and @supplier and (b) the creation of the link is
+ * properly synchronized with runtime PM.
+ *
+ * To that end, setting the DEVICE_LINK_PM_RUNTIME flag will cause the runtime
+ * PM framework to take the link into account.  Moreover, if the
+ * DEVICE_LINK_RPM_ACTIVE flag is set in addition to it, the supplier devices
+ * will be forced into the active metastate and reference-counted upon the
+ * creation of the link.  If DEVICE_LINK_PM_RUNTIME is not set,
+ * DEVICE_LINK_RPM_ACTIVE will be ignored.
  *
  * If the DEVICE_LINK_AUTOREMOVE is set, the link will be removed automatically
  * when the consumer device driver unbinds from it.  The combination of both
@@ -154,10 +162,19 @@ struct device_link *device_link_add(stru
 		if (link->consumer == consumer)
 			goto out;
 
-	link = kmalloc(sizeof(*link), GFP_KERNEL);
+	link = kzalloc(sizeof(*link), GFP_KERNEL);
 	if (!link)
 		goto out;
 
+	if ((flags & DEVICE_LINK_PM_RUNTIME) && (flags & DEVICE_LINK_RPM_ACTIVE)) {
+		if (pm_runtime_get_sync(supplier) < 0) {
+			pm_runtime_put_noidle(supplier);
+			kfree(link);
+			link = NULL;
+			goto out;
+		}
+		link->rpm_active = true;
+	}
 	get_device(supplier);
 	link->supplier = supplier;
 	INIT_LIST_HEAD(&link->s_node);
Index: linux-pm/drivers/base/dd.c
===================================================================
--- linux-pm.orig/drivers/base/dd.c
+++ linux-pm/drivers/base/dd.c
@@ -498,6 +498,7 @@ int driver_probe_device(struct device_dr
 	pr_debug("bus: '%s': %s: matched device %s with driver %s\n",
 		 drv->bus->name, __func__, dev_name(dev), drv->name);
 
+	pm_runtime_get_suppliers(dev);
 	if (dev->parent)
 		pm_runtime_get_sync(dev->parent);
 
@@ -508,6 +509,7 @@ int driver_probe_device(struct device_dr
 	if (dev->parent)
 		pm_runtime_put(dev->parent);
 
+	pm_runtime_put_suppliers(dev);
 	return ret;
 }
 
@@ -791,6 +793,7 @@ static void __device_release_driver(stru
 		}
 
 		pm_runtime_get_sync(dev);
+		pm_runtime_clean_up_links(dev);
 
 		driver_sysfs_remove(dev);
 
Index: linux-pm/drivers/base/power/runtime.c
===================================================================
--- linux-pm.orig/drivers/base/power/runtime.c
+++ linux-pm/drivers/base/power/runtime.c
@@ -12,6 +12,8 @@
 #include <linux/pm_runtime.h>
 #include <linux/pm_wakeirq.h>
 #include <trace/events/rpm.h>
+
+#include "../base.h"
 #include "power.h"
 
 typedef int (*pm_callback_t)(struct device *);
@@ -258,6 +260,42 @@ static int rpm_check_suspend_allowed(str
 	return retval;
 }
 
+static int rpm_get_suppliers(struct device *dev)
+{
+	struct device_link *link;
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node) {
+		int retval;
+
+		if (!(link->flags & DEVICE_LINK_PM_RUNTIME))
+			continue;
+
+		if (link->status == DEVICE_LINK_SUPPLIER_UNBIND ||
+		    link->rpm_active)
+			continue;
+
+		retval = pm_runtime_get_sync(link->supplier);
+		if (retval < 0) {
+			pm_runtime_put_noidle(link->supplier);
+			return retval;
+		}
+		link->rpm_active = true;
+	}
+	return 0;
+}
+
+static void rpm_put_suppliers(struct device *dev)
+{
+	struct device_link *link;
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node)
+		if (link->rpm_active &&
+		    link->status != DEVICE_LINK_SUPPLIER_UNBIND) {
+			pm_runtime_put(link->supplier);
+			link->rpm_active = false;
+		}
+}
+
 /**
  * __rpm_callback - Run a given runtime PM callback for a given device.
  * @cb: Runtime PM callback to run.
@@ -266,19 +304,55 @@ static int rpm_check_suspend_allowed(str
 static int __rpm_callback(int (*cb)(struct device *), struct device *dev)
 	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
-	int retval;
+	int retval, idx;
 
-	if (dev->power.irq_safe)
+	if (dev->power.irq_safe) {
 		spin_unlock(&dev->power.lock);
-	else
+	} else {
 		spin_unlock_irq(&dev->power.lock);
 
+		/*
+		 * Resume suppliers if necessary.
+		 *
+		 * The device's runtime PM status cannot change until this
+		 * routine returns, so it is safe to read the status outside of
+		 * the lock.
+		 */
+		if (dev->power.runtime_status == RPM_RESUMING) {
+			idx = device_links_read_lock();
+
+			retval = rpm_get_suppliers(dev);
+			if (retval)
+				goto fail;
+
+			device_links_read_unlock(idx);
+		}
+	}
+
 	retval = cb(dev);
 
-	if (dev->power.irq_safe)
+	if (dev->power.irq_safe) {
 		spin_lock(&dev->power.lock);
-	else
+	} else {
+		/*
+		 * If the device is suspending and the callback has returned
+		 * success, drop the usage counters of the suppliers that have
+		 * been reference counted on its resume.
+		 *
+		 * Do that if resume fails too.
+		 */
+		if ((dev->power.runtime_status == RPM_SUSPENDING && !retval)
+		    || (dev->power.runtime_status == RPM_RESUMING && retval)) {
+			idx = device_links_read_lock();
+
+ fail:
+			rpm_put_suppliers(dev);
+
+			device_links_read_unlock(idx);
+		}
+
 		spin_lock_irq(&dev->power.lock);
+	}
 
 	return retval;
 }
@@ -1447,6 +1521,79 @@ void pm_runtime_remove(struct device *de
 }
 
 /**
+ * pm_runtime_clean_up_links - Prepare links to consumers for driver removal.
+ * @dev: Device whose driver is going to be removed.
+ *
+ * Check links from this device to any consumers and if any of them have active
+ * runtime PM references to the device, drop the usage counter of the device
+ * (once per link).
+ *
+ * Links with the DEVICE_LINK_STATELESS flag set are ignored.
+ *
+ * Since the device is guaranteed to be runtime-active at the point this is
+ * called, nothing else needs to be done here.
+ *
+ * Moreover, this is called after device_links_busy() has returned 'false', so
+ * the status of each link is guaranteed to be DEVICE_LINK_SUPPLIER_UNBIND and
+ * therefore rpm_active can't be manipulated concurrently.
+ */
+void pm_runtime_clean_up_links(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_consumers, s_node) {
+		if (link->flags & DEVICE_LINK_STATELESS)
+			continue;
+
+		if (link->rpm_active) {
+			pm_runtime_put_noidle(dev);
+			link->rpm_active = false;
+		}
+	}
+
+	device_links_read_unlock(idx);
+}
+
+/**
+ * pm_runtime_get_suppliers - Resume and reference-count supplier devices.
+ * @dev: Consumer device.
+ */
+void pm_runtime_get_suppliers(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node)
+		if (link->flags & DEVICE_LINK_PM_RUNTIME)
+			pm_runtime_get_sync(link->supplier);
+
+	device_links_read_unlock(idx);
+}
+
+/**
+ * pm_runtime_put_suppliers - Drop references to supplier devices.
+ * @dev: Consumer device.
+ */
+void pm_runtime_put_suppliers(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links_to_suppliers, c_node)
+		if (link->flags & DEVICE_LINK_PM_RUNTIME)
+			pm_runtime_put(link->supplier);
+
+	device_links_read_unlock(idx);
+}
+
+/**
  * pm_runtime_force_suspend - Force a device into suspend state if needed.
  * @dev: Device to suspend.
  *
Index: linux-pm/include/linux/device.h
===================================================================
--- linux-pm.orig/include/linux/device.h
+++ linux-pm/include/linux/device.h
@@ -720,9 +720,13 @@ enum device_link_status {
  *
  * STATELESS: The core won't track the presence of supplier/consumer drivers.
  * AUTOREMOVE: Remove this link automatically on cunsumer driver unbind.
+ * PM_RUNTIME: If set, the runtime PM framework will use this link.
+ * RPM_ACTIVE: Run pm_runtime_get_sync() on the supplier during link creation.
  */
 #define DEVICE_LINK_STATELESS	(1 << 0)
 #define DEVICE_LINK_AUTOREMOVE	(1 << 1)
+#define DEVICE_LINK_PM_RUNTIME	(1 << 2)
+#define DEVICE_LINK_RPM_ACTIVE	(1 << 3)
 
 struct device_link {
 	struct device *supplier;
@@ -731,6 +735,7 @@ struct device_link {
 	struct list_head c_node;
 	enum device_link_status status;
 	u32 flags;
+	bool rpm_active;
 	spinlock_t lock;
 	struct rcu_head rcu_head;
 };
Index: linux-pm/include/linux/pm_runtime.h
===================================================================
--- linux-pm.orig/include/linux/pm_runtime.h
+++ linux-pm/include/linux/pm_runtime.h
@@ -55,6 +55,9 @@ extern unsigned long pm_runtime_autosusp
 extern void pm_runtime_update_max_time_suspended(struct device *dev,
 						 s64 delta_ns);
 extern void pm_runtime_set_memalloc_noio(struct device *dev, bool enable);
+extern void pm_runtime_clean_up_links(struct device *dev);
+extern void pm_runtime_get_suppliers(struct device *dev);
+extern void pm_runtime_put_suppliers(struct device *dev);
 
 static inline void pm_suspend_ignore_children(struct device *dev, bool enable)
 {
@@ -186,6 +189,9 @@ static inline unsigned long pm_runtime_a
 				struct device *dev) { return 0; }
 static inline void pm_runtime_set_memalloc_noio(struct device *dev,
 						bool enable){}
+static inline void pm_runtime_clean_up_links(struct device *dev) {}
+static inline void pm_runtime_get_suppliers(struct device *dev) {}
+static inline void pm_runtime_put_suppliers(struct device *dev) {}
 
 #endif /* !CONFIG_PM */
 

[Rebase][PATCH v4 5/5] PM / runtime: Optimize the use of device links

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

If the device has no links to suppliers that should be used for
runtime PM (links with DEVICE_LINK_PM_RUNTIME set), there is no
reason to walk the list of suppliers for that device during
runtime suspend and resume.

Add a simple mechanism to detect that case and possibly avoid the
extra unnecessary overhead.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/base/core.c          |   20 +++++++++++++-------
 drivers/base/power/runtime.c |   23 ++++++++++++++++++++---
 include/linux/pm.h           |    1 +
 include/linux/pm_runtime.h   |    4 ++++
 4 files changed, 38 insertions(+), 10 deletions(-)

Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -166,14 +166,17 @@ struct device_link *device_link_add(stru
 	if (!link)
 		goto out;
 
-	if ((flags & DEVICE_LINK_PM_RUNTIME) && (flags & DEVICE_LINK_RPM_ACTIVE)) {
-		if (pm_runtime_get_sync(supplier) < 0) {
-			pm_runtime_put_noidle(supplier);
-			kfree(link);
-			link = NULL;
-			goto out;
+	if (flags & DEVICE_LINK_PM_RUNTIME) {
+		if (flags & DEVICE_LINK_RPM_ACTIVE) {
+			if (pm_runtime_get_sync(supplier) < 0) {
+				pm_runtime_put_noidle(supplier);
+				kfree(link);
+				link = NULL;
+				goto out;
+			}
+			link->rpm_active = true;
 		}
-		link->rpm_active = true;
+		pm_runtime_new_link(consumer);
 	}
 	get_device(supplier);
 	link->supplier = supplier;
@@ -222,6 +225,9 @@ static void __device_link_del(struct dev
 	dev_info(link->consumer, "Dropping the link to %s\n",
 		 dev_name(link->supplier));
 
+	if (link->flags & DEVICE_LINK_PM_RUNTIME)
+		pm_runtime_drop_link(link->consumer);
+
 	list_del_rcu(&link->s_node);
 	list_del_rcu(&link->c_node);
 	call_srcu(&device_links_srcu, &link->rcu_head, __device_link_free_srcu);
Index: linux-pm/drivers/base/power/runtime.c
===================================================================
--- linux-pm.orig/drivers/base/power/runtime.c
+++ linux-pm/drivers/base/power/runtime.c
@@ -305,6 +305,7 @@ static int __rpm_callback(int (*cb)(stru
 	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
 	int retval, idx;
+	bool use_links = dev->power.links_count > 0;
 
 	if (dev->power.irq_safe) {
 		spin_unlock(&dev->power.lock);
@@ -318,7 +319,7 @@ static int __rpm_callback(int (*cb)(stru
 		 * routine returns, so it is safe to read the status outside of
 		 * the lock.
 		 */
-		if (dev->power.runtime_status == RPM_RESUMING) {
+		if (use_links && dev->power.runtime_status == RPM_RESUMING) {
 			idx = device_links_read_lock();
 
 			retval = rpm_get_suppliers(dev);
@@ -341,8 +342,9 @@ static int __rpm_callback(int (*cb)(stru
 		 *
 		 * Do that if resume fails too.
 		 */
-		if ((dev->power.runtime_status == RPM_SUSPENDING && !retval)
-		    || (dev->power.runtime_status == RPM_RESUMING && retval)) {
+		if (use_links
+		    && ((dev->power.runtime_status == RPM_SUSPENDING && !retval)
+		    || (dev->power.runtime_status == RPM_RESUMING && retval))) {
 			idx = device_links_read_lock();
 
  fail:
@@ -1593,6 +1595,21 @@ void pm_runtime_put_suppliers(struct dev
 	device_links_read_unlock(idx);
 }
 
+void pm_runtime_new_link(struct device *dev)
+{
+	spin_lock_irq(&dev->power.lock);
+	dev->power.links_count++;
+	spin_unlock_irq(&dev->power.lock);
+}
+
+void pm_runtime_drop_link(struct device *dev)
+{
+	spin_lock_irq(&dev->power.lock);
+	WARN_ON(dev->power.links_count == 0);
+	dev->power.links_count--;
+	spin_unlock_irq(&dev->power.lock);
+}
+
 /**
  * pm_runtime_force_suspend - Force a device into suspend state if needed.
  * @dev: Device to suspend.
Index: linux-pm/include/linux/pm.h
===================================================================
--- linux-pm.orig/include/linux/pm.h
+++ linux-pm/include/linux/pm.h
@@ -597,6 +597,7 @@ struct dev_pm_info {
 	unsigned int		use_autosuspend:1;
 	unsigned int		timer_autosuspends:1;
 	unsigned int		memalloc_noio:1;
+	unsigned int		links_count;
 	enum rpm_request	request;
 	enum rpm_status		runtime_status;
 	int			runtime_error;
Index: linux-pm/include/linux/pm_runtime.h
===================================================================
--- linux-pm.orig/include/linux/pm_runtime.h
+++ linux-pm/include/linux/pm_runtime.h
@@ -58,6 +58,8 @@ extern void pm_runtime_set_memalloc_noio
 extern void pm_runtime_clean_up_links(struct device *dev);
 extern void pm_runtime_get_suppliers(struct device *dev);
 extern void pm_runtime_put_suppliers(struct device *dev);
+extern void pm_runtime_new_link(struct device *dev);
+extern void pm_runtime_drop_link(struct device *dev);
 
 static inline void pm_suspend_ignore_children(struct device *dev, bool enable)
 {
@@ -192,6 +194,8 @@ static inline void pm_runtime_set_memall
 static inline void pm_runtime_clean_up_links(struct device *dev) {}
 static inline void pm_runtime_get_suppliers(struct device *dev) {}
 static inline void pm_runtime_put_suppliers(struct device *dev) {}
+static inline void pm_runtime_new_link(struct device *dev) {}
+static inline void pm_runtime_drop_link(struct device *dev) {}
 
 #endif /* !CONFIG_PM */
 

Re: [PATCH v4 0/5] Functional dependencies between devices

[Marek Szyprowski]

Hi Rafael,

On 2016-09-29 02:24, Rafael J. Wysocki wrote:
> Hi Everyone,
>
> On Thursday, September 08, 2016 11:25:44 PM Rafael J. Wysocki wrote:
>>> Hi Everyone,
>>>
>>> This is a refresh of the functional dependencies series that I posted last
>>> year and which has picked up by Marek quite recently.  For reference,
>>> appended is my introductory message sent previously (which may be
>>> slightly outdated now).
>>>
>>> As last time, the first patch rearranges the code around
>>> __device_release_driver() a bit to prepare it for the next one (it
>>> actually hasn't changed AFAICS).
>>>
>>> The second patch introduces the actual device links mechanics, but without
>>> system suspend/resume and runtime PM support which are added by the
>>> subsequent patches.
>>>
>>> Some bugs found by Marek during his work on these patches should be fixed
>>> here.  In particular, the endless recursion in device_reorder_to_tail()
>>> which simply was broken before.
>>>
>>> There are two additional patches to address the issue with runtime PM
>>> support
>>> that occured when runtime PM was disabled for some suppliers due to a PM
>>> sleep transition in progress.  Those patches simply make runtime PM
>>> helpers
>>> return 0 in that case which may be controversial, so please let me know if
>>> there are concerns about those.
>>>
>>> The way device_link_add() works is a bit different, as it takes an
>>> additional
>>> status argument now.  That makes it possible to create a link in any
>>> state,
>>> with extra care of course, and should address the problem pointed to by
>>> Lukas
>>> during the previous discussion.
>>>
>>> Also some comments from Tomeu have been addressed.
>> An update here.
>>
>> The first patch hasn't changed, so I'm resending it.
>>
>> The majority of changes in the other patches are in order to address Lukas'
>> comments.
>>
>> First off, I added a DEVICE_LINK_STATELESS flag that will prevent the driver
>> core from trying to maintain device links having it set.
>>
>> Also, the DEVICE_LINK_PERSISTENT flag was dropped (as link "persistence" is
>> the default behavior now) and there's a new one, DEVICE_LINK_AUTOREMOVE,
>> that will cause the driver core to remove the link on the consumer driver
>> unbind.
>>
>> Moreover, the code checks attempts to create a link between a parent and a
>> child device now and actively prevents that from happening.
>>
>> The changelog of the second patch has been updated as requested by Ulf.
>>
>> The third patch was updated to fix a bug related to the (previously missing)
>> clearing of power.direct_complete for supplier devices having consumers that
>> don't use direct_complete.
>>
>> The next two (runtime PM) patches turned out to be unnecessary, so I've
>> dropped them.
>>
>> The runtime PM patch [4/5] was reorganized somewhat to reduce the
>> indentation
>> level in there, but the code flow introduced by it is essentially the same
>> and the last patch was simply rebased on top of the new series.
> Time for another update. :-)
>
> Fewer changes this time, mostly to address issues found by Lukas and Marek.
>
> The most significant one is to make device_link_add() cope with the case when
> the consumer device has not been registered yet when it is called.  The
> supplier device still is required to be registered and the function will return
> NULL if that is not the case.
>
> Another significant change is in patch [4/5] that now makes the core apply
> pm_runtime_get_sync()/pm_runtime_put() to supplier devices around the probing
> of a consumer one (in analogy with the parent).

Thanks for the update! Updated version fixes all the remaining issues.

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [PATCH v4 0/5] Functional dependencies between devices

[Rafael J. Wysocki]

On Thursday, September 29, 2016 08:58:43 AM Marek Szyprowski wrote:
> Hi Rafael,
> 
> On 2016-09-29 02:24, Rafael J. Wysocki wrote:
> > Hi Everyone,
> >
> > On Thursday, September 08, 2016 11:25:44 PM Rafael J. Wysocki wrote:
> >>> Hi Everyone,
> >>>
> >>> This is a refresh of the functional dependencies series that I posted last
> >>> year and which has picked up by Marek quite recently.  For reference,
> >>> appended is my introductory message sent previously (which may be
> >>> slightly outdated now).
> >>>
> >>> As last time, the first patch rearranges the code around
> >>> __device_release_driver() a bit to prepare it for the next one (it
> >>> actually hasn't changed AFAICS).
> >>>
> >>> The second patch introduces the actual device links mechanics, but without
> >>> system suspend/resume and runtime PM support which are added by the
> >>> subsequent patches.
> >>>
> >>> Some bugs found by Marek during his work on these patches should be fixed
> >>> here.  In particular, the endless recursion in device_reorder_to_tail()
> >>> which simply was broken before.
> >>>
> >>> There are two additional patches to address the issue with runtime PM
> >>> support
> >>> that occured when runtime PM was disabled for some suppliers due to a PM
> >>> sleep transition in progress.  Those patches simply make runtime PM
> >>> helpers
> >>> return 0 in that case which may be controversial, so please let me know if
> >>> there are concerns about those.
> >>>
> >>> The way device_link_add() works is a bit different, as it takes an
> >>> additional
> >>> status argument now.  That makes it possible to create a link in any
> >>> state,
> >>> with extra care of course, and should address the problem pointed to by
> >>> Lukas
> >>> during the previous discussion.
> >>>
> >>> Also some comments from Tomeu have been addressed.
> >> An update here.
> >>
> >> The first patch hasn't changed, so I'm resending it.
> >>
> >> The majority of changes in the other patches are in order to address Lukas'
> >> comments.
> >>
> >> First off, I added a DEVICE_LINK_STATELESS flag that will prevent the driver
> >> core from trying to maintain device links having it set.
> >>
> >> Also, the DEVICE_LINK_PERSISTENT flag was dropped (as link "persistence" is
> >> the default behavior now) and there's a new one, DEVICE_LINK_AUTOREMOVE,
> >> that will cause the driver core to remove the link on the consumer driver
> >> unbind.
> >>
> >> Moreover, the code checks attempts to create a link between a parent and a
> >> child device now and actively prevents that from happening.
> >>
> >> The changelog of the second patch has been updated as requested by Ulf.
> >>
> >> The third patch was updated to fix a bug related to the (previously missing)
> >> clearing of power.direct_complete for supplier devices having consumers that
> >> don't use direct_complete.
> >>
> >> The next two (runtime PM) patches turned out to be unnecessary, so I've
> >> dropped them.
> >>
> >> The runtime PM patch [4/5] was reorganized somewhat to reduce the
> >> indentation
> >> level in there, but the code flow introduced by it is essentially the same
> >> and the last patch was simply rebased on top of the new series.
> > Time for another update. :-)
> >
> > Fewer changes this time, mostly to address issues found by Lukas and Marek.
> >
> > The most significant one is to make device_link_add() cope with the case when
> > the consumer device has not been registered yet when it is called.  The
> > supplier device still is required to be registered and the function will return
> > NULL if that is not the case.
> >
> > Another significant change is in patch [4/5] that now makes the core apply
> > pm_runtime_get_sync()/pm_runtime_put() to supplier devices around the probing
> > of a consumer one (in analogy with the parent).
> 
> Thanks for the update! Updated version fixes all the remaining issues.
> 
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>

Thanks Marek!

Re: [PATCH v4 0/5] Functional dependencies between devices

[Lukas Wunner]

[+cc Andreas Noever]

On Thu, Sep 29, 2016 at 02:24:58AM +0200, Rafael J. Wysocki wrote:
> On Thursday, September 08, 2016 11:25:44 PM Rafael J. Wysocki wrote:
> > > This is a refresh of the functional dependencies series that I posted
> > > last year and which has been picked up by Marek quite recently.

I've cooked up a patch to replace quirk_apple_wait_for_thunderbolt()
in drivers/pci/quirks.c with the "device links" functionality added by
Rafael's series quoted above.

The patch is included below.  I've also pushed a branch to GitHub which
comprises Rafael's series plus the patch on top:
https://github.com/l1k/linux/commits/device_links

One issue that cropped up is that the API does not provide public
functions to lock a device's link lists for traversal.  Thus when
iterating over the links and deleting them in the thunderbolt driver's
->remove hook, the list is completely unprotected.  It's not an issue
for this particular use case as noone else but this driver adds or
deletes links to the NHI, it just *looks* a bit fishy and there may
be other use cases where locking matters.  Maybe patch [2/5] should
export device_links_read_lock() / device_links_read_unlock()?

Apart from that, everything seems to work as it should:

On driver load:
[   13.829752] thunderbolt 0000:07:00.0: NHI initialized, starting thunderbolt
[...]
[   13.853747] pcieport 0000:06:03.0: Linked as a consumer to 0000:07:00.0
[   13.853749] pcieport 0000:06:04.0: Linked as a consumer to 0000:07:00.0
[   13.853751] pcieport 0000:06:05.0: Linked as a consumer to 0000:07:00.0
[   13.853753] pcieport 0000:06:06.0: Linked as a consumer to 0000:07:00.0

Those are the four hotplug ports on the controller.

On driver unload:
[   89.378691] pcieport 0000:06:03.0: Dropping the link to 0000:07:00.0
[   89.383346] pcieport 0000:06:04.0: Dropping the link to 0000:07:00.0
[   89.387977] pcieport 0000:06:05.0: Dropping the link to 0000:07:00.0
[   89.392589] pcieport 0000:06:06.0: Dropping the link to 0000:07:00.0
[...]
[   89.424511] thunderbolt 0000:07:00.0: shutdown

On resume from system sleep:
[  282.537470] ACPI: Waking up from system sleep state S3
[...]
[  282.625378] pcieport 0000:06:04.0: start waiting for 0000:07:00.0
[  282.625380] pcieport 0000:06:03.0: start waiting for 0000:07:00.0
[  282.625382] pcieport 0000:06:05.0: start waiting for 0000:07:00.0
[  282.625383] pcieport 0000:06:06.0: start waiting for 0000:07:00.0
[  282.656789] thunderbolt 0000:07:00.0: resuming...
[...]
[  283.500660] thunderbolt 0000:07:00.0: resume finished
[  283.500672] pcieport 0000:06:04.0:  done waiting for 0000:07:00.0
[  283.500673] pcieport 0000:06:03.0:  done waiting for 0000:07:00.0
[  283.500675] pcieport 0000:06:05.0:  done waiting for 0000:07:00.0
[  283.500677] pcieport 0000:06:06.0:  done waiting for 0000:07:00.0
[  283.564849] PM: noirq resume of devices complete after 971.845 msecs
[  283.564868] pciehp 0000:06:04.0:pcie204: Slot(4-1): Card present
[  283.564873] pciehp 0000:06:04.0:pcie204: Slot(4-1): Link Up

The "start waiting for" and "done waiting for" messages are debug
printk's that I had put in there.  I've also rebased and tested this
on my Thunderbolt runpm series and the only difference is that the
dpm_wait() is only executed for port 0000:06:04.0 (the one which
actually has a device connected), the three other hotplug ports use
direct_complete.

So Rafael's series is now also

Tested-by: Lukas Wunner <lukas@wunner.de>

though it should be noted that my patch does not make use of the optional
DEVICE_LINK_PM_RUNTIME flag introduced with patch [4/5].  (But I believe
Marek has tested that feature.)

Thanks,

Lukas

-- >8 --
Subject: [PATCH] thunderbolt: Use device links instead of PCI quirk

When resuming from system sleep, attached Thunderbolt devices are
inaccessible until the NHI has re-established PCI tunnels to them.  As a
consequence, the Thunderbolt controller's hotplug bridges (below which
the attached devices appear) must delay resuming until the NHI has
finished.  That requirement is not enforced by the PM core automatically
as it only guarantees correct resume ordering between parent and child,
and the NHI is not a parent of the hotplug bridges, but rather a niece.

So far we've open coded this requirement in a PCI quirk, which has the
following disadvantages:

- The code for the NHI resides in drivers/thunderbolt/, whereas the code
  for the hotplug bridges resides in drivers/pci/quirks.c, which may be
  surprising and non-obvious in particular for new contributors.

- Whenever support for an additional Thunderbolt controller is added,
  its PCI device ID needs to be amended in both places, which invites
  mistakes.  E.g. commit a42fb351ca1f ("thunderbolt: Allow loading of
  module on recent Apple MacBooks with thunderbolt 2 controller")
  relaxed the subvendor and subdevice ID in the NHI code but forgot to
  also change the hotplug bridge code.

- Since the PCI quirk cannot keep any state, it has to search for the
  NHI over and over again on every resume.

The PM core has just gained the ability to declare "device links", i.e.
dependencies between devices beyond the mere parent/child relationship.

Replace the PCI quirk with device links from the hotplug bridges to the
NHI.  The device links are set up when loading the thunderbolt driver
and torn down when it is removed.  The PM core takes care of everything
else.  As a bonus, the amount of code is reduced considerably.

Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Andreas Noever <andreas.noever@gmail.com>
Signed-off-by: Lukas Wunner <lukas@wunner.de>
---
 drivers/pci/quirks.c      | 56 -----------------------------------------------
 drivers/thunderbolt/nhi.c | 28 ++++++++++++++++++++++--
 drivers/thunderbolt/tb.h  |  1 +
 3 files changed, 27 insertions(+), 58 deletions(-)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 44e0ff3..9b78a03 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -3293,62 +3293,6 @@ static void quirk_apple_poweroff_thunderbolt(struct pci_dev *dev)
 DECLARE_PCI_FIXUP_SUSPEND_LATE(PCI_VENDOR_ID_INTEL,
 			       PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C,
 			       quirk_apple_poweroff_thunderbolt);
-
-/*
- * Apple: Wait for the thunderbolt controller to reestablish pci tunnels.
- *
- * During suspend the thunderbolt controller is reset and all pci
- * tunnels are lost. The NHI driver will try to reestablish all tunnels
- * during resume. We have to manually wait for the NHI since there is
- * no parent child relationship between the NHI and the tunneled
- * bridges.
- */
-static void quirk_apple_wait_for_thunderbolt(struct pci_dev *dev)
-{
-	struct pci_dev *sibling = NULL;
-	struct pci_dev *nhi = NULL;
-
-	if (!dmi_match(DMI_BOARD_VENDOR, "Apple Inc."))
-		return;
-	if (pci_pcie_type(dev) != PCI_EXP_TYPE_DOWNSTREAM)
-		return;
-	/*
-	 * Find the NHI and confirm that we are a bridge on the tb host
-	 * controller and not on a tb endpoint.
-	 */
-	sibling = pci_get_slot(dev->bus, 0x0);
-	if (sibling == dev)
-		goto out; /* we are the downstream bridge to the NHI */
-	if (!sibling || !sibling->subordinate)
-		goto out;
-	nhi = pci_get_slot(sibling->subordinate, 0x0);
-	if (!nhi)
-		goto out;
-	if (nhi->vendor != PCI_VENDOR_ID_INTEL
-		    || (nhi->device != PCI_DEVICE_ID_INTEL_LIGHT_RIDGE &&
-			nhi->device != PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C &&
-			nhi->device != PCI_DEVICE_ID_INTEL_FALCON_RIDGE_2C_NHI &&
-			nhi->device != PCI_DEVICE_ID_INTEL_FALCON_RIDGE_4C_NHI)
-		    || nhi->class != PCI_CLASS_SYSTEM_OTHER << 8)
-		goto out;
-	dev_info(&dev->dev, "quirk: waiting for thunderbolt to reestablish PCI tunnels...\n");
-	device_pm_wait_for_dev(&dev->dev, &nhi->dev);
-out:
-	pci_dev_put(nhi);
-	pci_dev_put(sibling);
-}
-DECLARE_PCI_FIXUP_RESUME_EARLY(PCI_VENDOR_ID_INTEL,
-			       PCI_DEVICE_ID_INTEL_LIGHT_RIDGE,
-			       quirk_apple_wait_for_thunderbolt);
-DECLARE_PCI_FIXUP_RESUME_EARLY(PCI_VENDOR_ID_INTEL,
-			       PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C,
-			       quirk_apple_wait_for_thunderbolt);
-DECLARE_PCI_FIXUP_RESUME_EARLY(PCI_VENDOR_ID_INTEL,
-			       PCI_DEVICE_ID_INTEL_FALCON_RIDGE_2C_BRIDGE,
-			       quirk_apple_wait_for_thunderbolt);
-DECLARE_PCI_FIXUP_RESUME_EARLY(PCI_VENDOR_ID_INTEL,
-			       PCI_DEVICE_ID_INTEL_FALCON_RIDGE_4C_BRIDGE,
-			       quirk_apple_wait_for_thunderbolt);
 #endif
 
 static void pci_do_fixups(struct pci_dev *dev, struct pci_fixup *f,
diff --git a/drivers/thunderbolt/nhi.c b/drivers/thunderbolt/nhi.c
index a8c2041..e57f2e4 100644
--- a/drivers/thunderbolt/nhi.c
+++ b/drivers/thunderbolt/nhi.c
@@ -533,6 +533,21 @@ static void nhi_shutdown(struct tb_nhi *nhi)
 	mutex_destroy(&nhi->lock);
 }
 
+static int nhi_device_link_add_cb(struct pci_dev *pdev, void *ptr)
+{
+	struct tb *tb = ptr;
+
+	/*
+	 * Let all downstream bridges of the switch depend on the NHI, except
+	 * for the NHI's parent bridge.  This forces them to dpm_wait() in the
+	 * ->resume_noirq phase until the NHI has re-established the tunnels.
+	 */
+	if (pdev->bus == tb->downstream0->bus && pdev != tb->downstream0)
+		device_link_add(&pdev->dev, &tb->nhi->pdev->dev,
+				DEVICE_LINK_NO_STATE, DEVICE_LINK_STATELESS);
+	return 0;
+}
+
 static int nhi_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 {
 	struct tb_nhi *nhi;
@@ -605,6 +620,10 @@ static int nhi_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	}
 	pci_set_drvdata(pdev, tb);
 
+	tb->downstream0 = pci_upstream_bridge(pdev);
+	if (tb->downstream0)
+		pci_walk_bus(tb->downstream0->bus, nhi_device_link_add_cb, tb);
+
 	return 0;
 }
 
@@ -612,14 +631,19 @@ static void nhi_remove(struct pci_dev *pdev)
 {
 	struct tb *tb = pci_get_drvdata(pdev);
 	struct tb_nhi *nhi = tb->nhi;
+	struct device_link *link, *ln;
+
+	list_for_each_entry_safe(link, ln, &pdev->dev.links_to_consumers, s_node)
+		device_link_del(link);
+
 	thunderbolt_shutdown_and_free(tb);
 	nhi_shutdown(nhi);
 }
 
 /*
  * The tunneled pci bridges are siblings of us. Use resume_noirq to reenable
- * the tunnels asap. A corresponding pci quirk blocks the downstream bridges
- * resume_noirq until we are done.
+ * the tunnels asap. Device links block the downstream bridges' resume_noirq
+ * until we are done.
  */
 static const struct dev_pm_ops nhi_pm_ops = {
 	.suspend_noirq = nhi_suspend_noirq,
diff --git a/drivers/thunderbolt/tb.h b/drivers/thunderbolt/tb.h
index 61d57ba..f9c45d5 100644
--- a/drivers/thunderbolt/tb.h
+++ b/drivers/thunderbolt/tb.h
@@ -106,6 +106,7 @@ struct tb {
 	struct workqueue_struct *wq; /* ordered workqueue for plug events */
 	struct tb_switch *root_switch;
 	struct list_head tunnel_list; /* list of active PCIe tunnels */
+	struct pci_dev *downstream0; /* downstream bridge to the NHI */
 	bool hotplug_active; /*
 			      * tb_handle_hotplug will stop progressing plug
 			      * events and exit if this is not set (it needs to
-- 
2.9.3

[PATCH v5 0/5] Functional dependencies between devices

[Rafael J. Wysocki]

Hi Everyone,

> On Thursday, September 08, 2016 11:25:44 PM Rafael J. Wysocki wrote:
> 

[cut]

> 
> Time for another update. :-)
> 
> Fewer changes this time, mostly to address issues found by Lukas and Marek.
> 
> The most significant one is to make device_link_add() cope with the case
> when
> the consumer device has not been registered yet when it is called.  The
> supplier device still is required to be registered and the function will
> return NULL if that is not the case.
> 
> Another significant change is in patch [4/5] that now makes the core apply
> pm_runtime_get_sync()/pm_runtime_put() to supplier devices around the
> probing of a consumer one (in analogy with the parent).

One more update after some conversations during LinuxCon Europe.

The main point was to make it possible for device_link_add() to figure out the
initial state of the link instead of expecting the caller to provide it which
might not be reliable enough in general.

In this version device_link_add() takes three arguments, the supplier and
consumer pointers and flags and it sets the correct initial state of the link
automatically (unless invoked with the "stateless" flag, of course).  The cost
is one additional field in struct device (I moved all of the links-related
fields in struct device to a separate sub-structure while at it) to track
the "driver presence status" of the device (to be used by device_link_add()).

In addition to that, the links list walks in the core.c and dd.c code are
under the device links mutex now, so the iternal link spinlock is not needed
any more and I have renamed symbols to distinguish between flags, link states
and device "driver presence statuses".

More information is there in the changelogs.

Thanks,
Rafael

[Resend][PATCH v5 1/5] driver core: Add a wrapper around __device_release_driver()

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Add an internal wrapper around __device_release_driver() that will
acquire device locks and do the necessary checks before calling it.

The next patch will make use of it.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/base/dd.c | 30 ++++++++++++++++++------------
 1 file changed, 18 insertions(+), 12 deletions(-)

diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index 16688f50729c..d9e76e9205c7 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -796,6 +796,22 @@ static void __device_release_driver(struct device *dev)
 	}
 }
 
+static void device_release_driver_internal(struct device *dev,
+					   struct device_driver *drv,
+					   struct device *parent)
+{
+	if (parent)
+		device_lock(parent);
+
+	device_lock(dev);
+	if (!drv || drv == dev->driver)
+		__device_release_driver(dev);
+
+	device_unlock(dev);
+	if (parent)
+		device_unlock(parent);
+}
+
 /**
  * device_release_driver - manually detach device from driver.
  * @dev: device.
@@ -810,9 +826,7 @@ void device_release_driver(struct device *dev)
 	 * within their ->remove callback for the same device, they
 	 * will deadlock right here.
 	 */
-	device_lock(dev);
-	__device_release_driver(dev);
-	device_unlock(dev);
+	device_release_driver_internal(dev, NULL, NULL);
 }
 EXPORT_SYMBOL_GPL(device_release_driver);
 
@@ -837,15 +851,7 @@ void driver_detach(struct device_driver *drv)
 		dev = dev_prv->device;
 		get_device(dev);
 		spin_unlock(&drv->p->klist_devices.k_lock);
-
-		if (dev->parent)	/* Needed for USB */
-			device_lock(dev->parent);
-		device_lock(dev);
-		if (dev->driver == drv)
-			__device_release_driver(dev);
-		device_unlock(dev);
-		if (dev->parent)
-			device_unlock(dev->parent);
+		device_release_driver_internal(dev, drv, dev->parent);
 		put_device(dev);
 	}
 }

[PATCH v5 2/5] driver core: Functional dependencies tracking support

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Currently, there is a problem with taking functional dependencies
between devices into account.

What I mean by a "functional dependency" is when the driver of device
B needs device A to be functional and (generally) its driver to be
present in order to work properly.  This has certain consequences
for power management (suspend/resume and runtime PM ordering) and
shutdown ordering of these devices.  In general, it also implies that
the driver of A needs to be working for B to be probed successfully
and it cannot be unbound from the device before the B's driver.

Support for representing those functional dependencies between
devices is added here to allow the driver core to track them and act
on them in certain cases where applicable.

The argument for doing that in the driver core is that there are
quite a few distinct use cases involving device dependencies, they
are relatively hard to get right in a driver (if one wants to
address all of them properly) and it only gets worse if multiplied
by the number of drivers potentially needing to do it.  Morever, at
least one case (asynchronous system suspend/resume) cannot be handled
in a single driver at all, because it requires the driver of A to
wait for B to suspend (during system suspend) and the driver of B to
wait for A to resume (during system resume).

For this reason, represent dependencies between devices as "links",
with the help of struct device_link objects each containing pointers
to the "linked" devices, a list node for each of them, status
information, flags, and an RCU head for synchronization.

Also add two new list heads, representing the lists of links to the
devices that depend on the given one (consumers) and to the devices
depended on by it (suppliers), and a "driver presence status" field
(needed for figuring out initial states of device links) to struct
device.

The entire data structure consisting of all of the lists of link
objects for all devices is protected by a mutex (for link object
addition/removal and for list walks during device driver probing
and removal) and by SRCU (for list walking in other case that will
be introduced by subsequent change sets).  In addition, each link
object has an internal status field whose value reflects whether or
not drivers are bound to the devices pointed to by the link or
probing/removal of their drivers is in progress etc.  That field
is only modified under the device links mutex, but it may be read
outside of it in some cases (introduced by subsequent change sets),
so modifications of it are annotated with WRITE_ONCE().

New links are added by calling device_link_add() which takes three
arguments: pointers to the devices in question and flags.  In
particular, if DL_FLAG_STATELESS is set in the flags, the link status
is not to be taken into account for this link and the driver core
will not manage it.  In turn, if DL_FLAG_AUTOREMOVE is set in the
flags, the driver core will remove the link automatically when the
consumer device driver unbinds from it.

One of the actions carried out by device_link_add() is to reorder
the lists used for device shutdown and system suspend/resume to
put the consumer device along with all of its children and all of
its consumers (and so on, recursively) to the ends of those lists
in order to ensure the right ordering between all of the supplier
and consumer devices.

For this reason, it is not possible to create a link between two
devices if the would-be supplier device already depends on the
would-be consumer device as either a direct descendant of it or a
consumer of one of its direct descendants or one of its consumers
and so on.

There are two types of link objects, persistent and non-persistent.
The persistent ones stay around until one of the target devices is
deleted, while the non-persistent ones are removed automatically when
the consumer driver unbinds from its device (ie. they are assumed to
be valid only as long as the consumer device has a driver bound to
it).  Persistent links are created by default and non-persistent
links are created when the DL_FLAG_AUTOREMOVE flag is passed
to device_link_add().

Both persistent and non-persistent device links can be deleted
with an explicit call to device_link_del().

Links created without the DL_FLAG_STATELESS flag set are managed
by the driver core using a simple state machine.  There are 5 states
each link can be in: DORMANT (unused), AVAILABLE (the supplier driver
is present and functional), CONSUMER_PROBE (the consumer driver is
probing), ACTIVE (both supplier and consumer drivers are present and
functional), and SUPPLIER_UNBIND (the supplier driver is unbinding).
The driver core updates the link state automatically depending on
what happens to the linked devices and for each link state specific
actions are taken in addition to that.

For example, if the supplier driver unbinds from its device, the
driver core will also unbind the drivers of all of its consumers
automatically under the assumption that they cannot function
properly without the supplier.  Analogously, the driver core will
only allow the consumer driver to bind to its device if the
supplier driver is present and functional (ie. the link is in
the AVAILABLE state).  If that's not the case, it will rely on
the existing deferred probing mechanism to wait for the supplier
driver to become available.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---

v4 -> v5:

- Redefine device_link_add() to take three arguments, the supplier and consumer
  pointers and flags, and to figure out the initial link state automatically.

- Move the links-related fields in struct device to a separate sub-structure
  and add a "driver presence tracking" field to it (to help device_link_add()
  to do its job).

- Modify device_links_check_suppliers(), device_links_driver_bound(),
  device_links_no_driver(), device_links_driver_cleanup(), device_links_busy(),
  and device_links_unbind_consumers() to walk link lists under device_links_lock
  (to make the new "driver presence tracking" mechanism work reliably).

- Drop the (not necessary any more) spinlock from struct device_link.

- Rename symbols to better reflect their purpose (flags vs link states etc).

v3 -> v4:

- Add the in_dpm_list field to struct dev_pm_info and use it for checking
  if the device has been added to dpm_list already, which is needed for
  handling the "not registered consumer" case in device_link_add().

- Rework device_link_add() to handle consumer devices that have not been
  registered before calling it.

- Drop the parent check from device_link_add().

- Rename device_links_driver_gone() to device_links_driver_cleanup()

- Rearrange device_links_unbind_consumers() to avoid one spin_unlock()
  instance (which wasn't necessary).

- Introduce device_links_purge() to delete links from a device going away
  and make device_del() call it (instead of running links-related code
  directly).

- Move the invocation of device_links_ckeck_suppliers() to really_probe().

- Change the description of the DEVICE_LINK_STATELESS flag.

---
 drivers/base/base.h        |   13 +
 drivers/base/core.c        |  491 +++++++++++++++++++++++++++++++++++++++++++++
 drivers/base/dd.c          |   41 +++
 drivers/base/power/main.c  |    2 
 drivers/base/power/power.h |   10 
 include/linux/device.h     |   78 +++++++
 include/linux/pm.h         |    1 
 7 files changed, 631 insertions(+), 5 deletions(-)

Index: linux-pm/drivers/base/base.h
===================================================================
--- linux-pm.orig/drivers/base/base.h
+++ linux-pm/drivers/base/base.h
@@ -107,6 +107,9 @@ extern void bus_remove_device(struct dev
 
 extern int bus_add_driver(struct device_driver *drv);
 extern void bus_remove_driver(struct device_driver *drv);
+extern void device_release_driver_internal(struct device *dev,
+					   struct device_driver *drv,
+					   struct device *parent);
 
 extern void driver_detach(struct device_driver *drv);
 extern int driver_probe_device(struct device_driver *drv, struct device *dev);
@@ -152,3 +155,13 @@ extern int devtmpfs_init(void);
 #else
 static inline int devtmpfs_init(void) { return 0; }
 #endif
+
+/* Device links support */
+extern int device_links_read_lock(void);
+extern void device_links_read_unlock(int idx);
+extern int device_links_check_suppliers(struct device *dev);
+extern void device_links_driver_bound(struct device *dev);
+extern void device_links_driver_cleanup(struct device *dev);
+extern void device_links_no_driver(struct device *dev);
+extern bool device_links_busy(struct device *dev);
+extern void device_links_unbind_consumers(struct device *dev);
Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -44,6 +44,492 @@ static int __init sysfs_deprecated_setup
 early_param("sysfs.deprecated", sysfs_deprecated_setup);
 #endif
 
+/* Device links support. */
+
+static DEFINE_MUTEX(device_links_lock);
+DEFINE_STATIC_SRCU(device_links_srcu);
+
+int device_links_read_lock(void)
+{
+	return srcu_read_lock(&device_links_srcu);
+}
+
+void device_links_read_unlock(int idx)
+{
+	return srcu_read_unlock(&device_links_srcu, idx);
+}
+
+/**
+ * device_is_dependent - Check if one device depends on another one
+ * @dev: Device to check dependencies for.
+ * @target: Device to check against.
+ *
+ * Check if @target depends on @dev or any device dependent on it (its child or
+ * its consumer etc).  Return 1 if that is the case or 0 otherwise.
+ */
+static int device_is_dependent(struct device *dev, void *target)
+{
+	struct device_link *link;
+	int ret;
+
+	if (WARN_ON(dev == target))
+		return 1;
+
+	ret = device_for_each_child(dev, target, device_is_dependent);
+	if (ret)
+		return ret;
+
+	list_for_each_entry(link, &dev->links.consumers, s_node) {
+		if (WARN_ON(link->consumer == target))
+			return 1;
+
+		ret = device_is_dependent(link->consumer, target);
+		if (ret)
+			break;
+	}
+	return ret;
+}
+
+static int device_reorder_to_tail(struct device *dev, void *not_used)
+{
+	struct device_link *link;
+
+	/*
+	 * Devices that have not been registered yet will be put to the ends
+	 * of the lists during the registration, so skip them here.
+	 */
+	if (device_is_registered(dev))
+		devices_kset_move_last(dev);
+
+	if (device_pm_initialized(dev))
+		device_pm_move_last(dev);
+
+	device_for_each_child(dev, NULL, device_reorder_to_tail);
+	list_for_each_entry(link, &dev->links.consumers, s_node)
+		device_reorder_to_tail(link->consumer, NULL);
+
+	return 0;
+}
+
+/**
+ * device_link_add - Create a link between two devices.
+ * @consumer: Consumer end of the link.
+ * @supplier: Supplier end of the link.
+ * @flags: Link flags.
+ *
+ * If the DL_FLAG_AUTOREMOVE is set, the link will be removed automatically
+ * when the consumer device driver unbinds from it.  The combination of both
+ * DL_FLAG_AUTOREMOVE and DL_FLAG_STATELESS set is invalid and will cause NULL
+ * to be returned.
+ *
+ * A side effect of the link creation is re-ordering of dpm_list and the
+ * devices_kset list by moving the consumer device and all devices depending
+ * on it to the ends of these lists (that does not happen to devices that have
+ * not been registered when this function is called).
+ *
+ * The supplier device is required to be registered when this function is called
+ * and NULL will be returned if that is not the case.  The consumer device need
+ * not be registerd, however.
+ */
+struct device_link *device_link_add(struct device *consumer,
+				    struct device *supplier, u32 flags)
+{
+	struct device_link *link;
+
+	if (!consumer || !supplier ||
+	    ((flags & DL_FLAG_STATELESS) && (flags & DL_FLAG_AUTOREMOVE)))
+		return NULL;
+
+	mutex_lock(&device_links_lock);
+	device_pm_lock();
+
+	/*
+	 * If the supplier has not been fully registered yet or there is a
+	 * reverse dependency between the consumer and the supplier already in
+	 * the graph, return NULL.
+	 */
+	if (!device_pm_initialized(supplier)
+	    || device_is_dependent(consumer, supplier)) {
+		link = NULL;
+		goto out;
+	}
+
+	list_for_each_entry(link, &supplier->links.consumers, s_node)
+		if (link->consumer == consumer)
+			goto out;
+
+	link = kmalloc(sizeof(*link), GFP_KERNEL);
+	if (!link)
+		goto out;
+
+	get_device(supplier);
+	link->supplier = supplier;
+	INIT_LIST_HEAD(&link->s_node);
+	get_device(consumer);
+	link->consumer = consumer;
+	INIT_LIST_HEAD(&link->c_node);
+	link->flags = flags;
+
+	/* Deterine the initial link state. */
+	if (flags & DL_FLAG_STATELESS) {
+		link->status = DL_STATE_NONE;
+	} else {
+		switch (supplier->links.status) {
+		case DL_DEV_DRIVER_BOUND:
+			switch (consumer->links.status) {
+			case DL_DEV_PROBING:
+				link->status = DL_STATE_CONSUMER_PROBE;
+				break;
+			case DL_DEV_DRIVER_BOUND:
+				link->status = DL_STATE_ACTIVE;
+				break;
+			default:
+				link->status = DL_STATE_AVAILABLE;
+				break;
+			}
+			break;
+		case DL_DEV_UNBINDING:
+			link->status = DL_STATE_SUPPLIER_UNBIND;
+			break;
+		default:
+			link->status = DL_STATE_DORMANT;
+			break;
+		}
+	}
+
+	/*
+	 * Move the consumer and all of the devices depending on it to the end
+	 * of dpm_list and the devices_kset list.
+	 *
+	 * It is necessary to hold dpm_list locked throughout all that or else
+	 * we may end up suspending with a wrong ordering of it.
+	 */
+	device_reorder_to_tail(consumer, NULL);
+
+	list_add_tail_rcu(&link->s_node, &supplier->links.consumers);
+	list_add_tail_rcu(&link->c_node, &consumer->links.suppliers);
+
+	dev_info(consumer, "Linked as a consumer to %s\n", dev_name(supplier));
+
+ out:
+	device_pm_unlock();
+	mutex_unlock(&device_links_lock);
+	return link;
+}
+EXPORT_SYMBOL_GPL(device_link_add);
+
+static void __device_link_free_srcu(struct rcu_head *rhead)
+{
+	struct device_link *link;
+
+	link = container_of(rhead, struct device_link, rcu_head);
+	put_device(link->consumer);
+	put_device(link->supplier);
+	kfree(link);
+}
+
+static void __device_link_del(struct device_link *link)
+{
+	dev_info(link->consumer, "Dropping the link to %s\n",
+		 dev_name(link->supplier));
+
+	list_del_rcu(&link->s_node);
+	list_del_rcu(&link->c_node);
+	call_srcu(&device_links_srcu, &link->rcu_head, __device_link_free_srcu);
+}
+
+/**
+ * device_link_del - Delete a link between two devices.
+ * @link: Device link to delete.
+ *
+ * The caller must ensure proper synchronization of this function with runtime
+ * PM.
+ */
+void device_link_del(struct device_link *link)
+{
+	mutex_lock(&device_links_lock);
+	device_pm_lock();
+	__device_link_del(link);
+	device_pm_unlock();
+	mutex_unlock(&device_links_lock);
+}
+EXPORT_SYMBOL_GPL(device_link_del);
+
+static void device_links_missing_supplier(struct device *dev)
+{
+	struct device_link *link;
+
+	list_for_each_entry(link, &dev->links.suppliers, c_node)
+		if (link->status == DL_STATE_CONSUMER_PROBE)
+			WRITE_ONCE(link->status, DL_STATE_AVAILABLE);
+}
+
+/**
+ * device_links_check_suppliers - Check presence of supplier drivers.
+ * @dev: Consumer device.
+ *
+ * Check links from this device to any suppliers.  Walk the list of the device's
+ * links to suppliers and see if all of them are available.  If not, simply
+ * return -EPROBE_DEFER.
+ *
+ * We need to guarantee that the supplier will not go away after the check has
+ * been positive here.  It only can go away in __device_release_driver() and
+ * that function  checks the device's links to consumers.  This means we need to
+ * mark the link as "consumer probe in progress" to make the supplier removal
+ * wait for us to complete (or bad things may happen).
+ *
+ * Links with the DL_FLAG_STATELESS flag set are ignored.
+ */
+int device_links_check_suppliers(struct device *dev)
+{
+	struct device_link *link;
+	int ret = 0;
+
+	mutex_lock(&device_links_lock);
+
+	list_for_each_entry(link, &dev->links.suppliers, c_node) {
+		if (link->flags & DL_FLAG_STATELESS)
+			continue;
+
+		if (link->status != DL_STATE_AVAILABLE) {
+			device_links_missing_supplier(dev);
+			ret = -EPROBE_DEFER;
+			break;
+		}
+		WRITE_ONCE(link->status, DL_STATE_CONSUMER_PROBE);
+	}
+	dev->links.status = DL_DEV_PROBING;
+
+	mutex_unlock(&device_links_lock);
+	return ret;
+}
+
+/**
+ * device_links_driver_bound - Update device links after probing its driver.
+ * @dev: Device to update the links for.
+ *
+ * The probe has been successful, so update links from this device to any
+ * consumers by changing their status to "available".
+ *
+ * Also change the status of @dev's links to suppliers to "active".
+ *
+ * Links with the DL_FLAG_STATELESS flag set are ignored.
+ */
+void device_links_driver_bound(struct device *dev)
+{
+	struct device_link *link;
+
+	mutex_lock(&device_links_lock);
+
+	list_for_each_entry(link, &dev->links.consumers, s_node) {
+		if (link->flags & DL_FLAG_STATELESS)
+			continue;
+
+		WARN_ON(link->status != DL_STATE_DORMANT);
+		WRITE_ONCE(link->status, DL_STATE_AVAILABLE);
+	}
+
+	list_for_each_entry(link, &dev->links.suppliers, c_node) {
+		if (link->flags & DL_FLAG_STATELESS)
+			continue;
+
+		WARN_ON(link->status != DL_STATE_CONSUMER_PROBE);
+		WRITE_ONCE(link->status, DL_STATE_ACTIVE);
+	}
+
+	dev->links.status = DL_DEV_DRIVER_BOUND;
+
+	mutex_unlock(&device_links_lock);
+}
+
+/**
+ * __device_links_no_driver - Update links of a device without a driver.
+ * @dev: Device without a drvier.
+ *
+ * Delete all non-persistent links from this device to any suppliers.
+ *
+ * Persistent links stay around, but their status is changed to "available",
+ * unless they already are in the "supplier unbind in progress" state in which
+ * case they need not be updated.
+ *
+ * Links with the DL_FLAG_STATELESS flag set are ignored.
+ */
+static void __device_links_no_driver(struct device *dev)
+{
+	struct device_link *link, *ln;
+
+	list_for_each_entry_safe_reverse(link, ln, &dev->links.suppliers, c_node) {
+		if (link->flags & DL_FLAG_STATELESS)
+			continue;
+
+		if (link->flags & DL_FLAG_AUTOREMOVE)
+			__device_link_del(link);
+		else if (link->status != DL_STATE_SUPPLIER_UNBIND)
+			WRITE_ONCE(link->status, DL_STATE_AVAILABLE);
+	}
+
+	dev->links.status = DL_DEV_NO_DRIVER;
+}
+
+void device_links_no_driver(struct device *dev)
+{
+	mutex_lock(&device_links_lock);
+	__device_links_no_driver(dev);
+	mutex_unlock(&device_links_lock);
+}
+
+/**
+ * device_links_driver_cleanup - Update links after driver removal.
+ * @dev: Device whose driver has just gone away.
+ *
+ * Update links to consumers for @dev by changing their status to "dormant" and
+ * invoke %__device_links_no_driver() to update links to suppliers for it as
+ * appropriate.
+ *
+ * Links with the DL_FLAG_STATELESS flag set are ignored.
+ */
+void device_links_driver_cleanup(struct device *dev)
+{
+	struct device_link *link;
+
+	mutex_lock(&device_links_lock);
+
+	list_for_each_entry(link, &dev->links.consumers, s_node) {
+		if (link->flags & DL_FLAG_STATELESS)
+			continue;
+
+		WARN_ON(link->flags & DL_FLAG_AUTOREMOVE);
+		WARN_ON(link->status != DL_STATE_SUPPLIER_UNBIND);
+		WRITE_ONCE(link->status, DL_STATE_DORMANT);
+	}
+
+	__device_links_no_driver(dev);
+
+	mutex_unlock(&device_links_lock);
+}
+
+/**
+ * device_links_busy - Check if there are any busy links to consumers.
+ * @dev: Device to check.
+ *
+ * Check each consumer of the device and return 'true' if its link's status
+ * is one of "consumer probe" or "active" (meaning that the given consumer is
+ * probing right now or its driver is present).  Otherwise, change the link
+ * state to "supplier unbind" to prevent the consumer from being probed
+ * successfully going forward.
+ *
+ * Return 'false' if there are no probing or active consumers.
+ *
+ * Links with the DL_FLAG_STATELESS flag set are ignored.
+ */
+bool device_links_busy(struct device *dev)
+{
+	struct device_link *link;
+	bool ret = false;
+
+	mutex_lock(&device_links_lock);
+
+	list_for_each_entry(link, &dev->links.consumers, s_node) {
+		if (link->flags & DL_FLAG_STATELESS)
+			continue;
+
+		if (link->status == DL_STATE_CONSUMER_PROBE
+		    || link->status == DL_STATE_ACTIVE) {
+			ret = true;
+			break;
+		}
+		WRITE_ONCE(link->status, DL_STATE_SUPPLIER_UNBIND);
+	}
+
+	dev->links.status = DL_DEV_UNBINDING;
+
+	mutex_unlock(&device_links_lock);
+	return ret;
+}
+
+/**
+ * device_links_unbind_consumers - Force unbind consumers of the given device.
+ * @dev: Device to unbind the consumers of.
+ *
+ * Walk the list of links to consumers for @dev and if any of them is in the
+ * "consumer probe" state, wait for all device probes in progress to complete
+ * and start over.
+ *
+ * If that's not the case, change the status of the link to "supplier unbind"
+ * and check if the link was in the "active" state.  If so, force the consumer
+ * driver to unbind and start over (the consumer will not re-probe as we have
+ * changed the state of the link already).
+ *
+ * Links with the DL_FLAG_STATELESS flag set are ignored.
+ */
+void device_links_unbind_consumers(struct device *dev)
+{
+	struct device_link *link;
+
+ start:
+	mutex_lock(&device_links_lock);
+
+	list_for_each_entry(link, &dev->links.consumers, s_node) {
+		enum device_link_state status;
+
+		if (link->flags & DL_FLAG_STATELESS)
+			continue;
+
+		status = link->status;
+		if (status == DL_STATE_CONSUMER_PROBE) {
+			mutex_unlock(&device_links_lock);
+
+			wait_for_device_probe();
+			goto start;
+		}
+		WRITE_ONCE(link->status, DL_STATE_SUPPLIER_UNBIND);
+		if (status == DL_STATE_ACTIVE) {
+			struct device *consumer = link->consumer;
+
+			get_device(consumer);
+
+			mutex_unlock(&device_links_lock);
+
+			device_release_driver_internal(consumer, NULL,
+						       consumer->parent);
+			put_device(consumer);
+			goto start;
+		}
+	}
+
+	mutex_unlock(&device_links_lock);
+}
+
+/**
+ * device_links_purge - Delete existing links to other devices.
+ * @dev: Target device.
+ */
+static void device_links_purge(struct device *dev)
+{
+	struct device_link *link, *ln;
+
+	/*
+	 * Delete all of the remaining links from this device to any other
+	 * devices (either consumers or suppliers).
+	 */
+	mutex_lock(&device_links_lock);
+
+	list_for_each_entry_safe_reverse(link, ln, &dev->links.suppliers, c_node) {
+		WARN_ON(link->status == DL_STATE_ACTIVE);
+		__device_link_del(link);
+	}
+
+	list_for_each_entry_safe_reverse(link, ln, &dev->links.consumers, s_node) {
+		WARN_ON(link->status != DL_STATE_DORMANT &&
+			link->status != DL_STATE_NONE);
+		__device_link_del(link);
+	}
+
+	mutex_unlock(&device_links_lock);
+}
+
+/* Device links support end. */
+
 int (*platform_notify)(struct device *dev) = NULL;
 int (*platform_notify_remove)(struct device *dev) = NULL;
 static struct kobject *dev_kobj;
@@ -711,6 +1197,9 @@ void device_initialize(struct device *de
 #ifdef CONFIG_GENERIC_MSI_IRQ
 	INIT_LIST_HEAD(&dev->msi_list);
 #endif
+	INIT_LIST_HEAD(&dev->links.consumers);
+	INIT_LIST_HEAD(&dev->links.suppliers);
+	dev->links.status = DL_DEV_NO_DRIVER;
 }
 EXPORT_SYMBOL_GPL(device_initialize);
 
@@ -1240,6 +1729,8 @@ void device_del(struct device *dev)
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DEL_DEVICE, dev);
+
+	device_links_purge(dev);
 	dpm_sysfs_remove(dev);
 	if (parent)
 		klist_del(&dev->p->knode_parent);
Index: linux-pm/drivers/base/dd.c
===================================================================
--- linux-pm.orig/drivers/base/dd.c
+++ linux-pm/drivers/base/dd.c
@@ -249,6 +249,7 @@ static void driver_bound(struct device *
 		 __func__, dev_name(dev));
 
 	klist_add_tail(&dev->p->knode_driver, &dev->driver->p->klist_devices);
+	device_links_driver_bound(dev);
 
 	device_pm_check_callbacks(dev);
 
@@ -341,6 +342,10 @@ static int really_probe(struct device *d
 		return ret;
 	}
 
+	ret = device_links_check_suppliers(dev);
+	if (ret)
+		return ret;
+
 	atomic_inc(&probe_count);
 	pr_debug("bus: '%s': %s: probing driver %s with device %s\n",
 		 drv->bus->name, __func__, drv->name, dev_name(dev));
@@ -399,6 +404,7 @@ probe_failed:
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
 pinctrl_bind_failed:
+	device_links_no_driver(dev);
 	devres_release_all(dev);
 	driver_sysfs_remove(dev);
 	dev->driver = NULL;
@@ -756,7 +762,7 @@ EXPORT_SYMBOL_GPL(driver_attach);
  * __device_release_driver() must be called with @dev lock held.
  * When called for a USB interface, @dev->parent lock must be held as well.
  */
-static void __device_release_driver(struct device *dev)
+static void __device_release_driver(struct device *dev, struct device *parent)
 {
 	struct device_driver *drv;
 
@@ -765,6 +771,25 @@ static void __device_release_driver(stru
 		if (driver_allows_async_probing(drv))
 			async_synchronize_full();
 
+		while (device_links_busy(dev)) {
+			device_unlock(dev);
+			if (parent)
+				device_unlock(parent);
+
+			device_links_unbind_consumers(dev);
+			if (parent)
+				device_lock(parent);
+
+			device_lock(dev);
+			/*
+			 * A concurrent invocation of the same function might
+			 * have released the driver successfully while this one
+			 * was waiting, so check for that.
+			 */
+			if (dev->driver != drv)
+				return;
+		}
+
 		pm_runtime_get_sync(dev);
 
 		driver_sysfs_remove(dev);
@@ -780,6 +805,8 @@ static void __device_release_driver(stru
 			dev->bus->remove(dev);
 		else if (drv->remove)
 			drv->remove(dev);
+
+		device_links_driver_cleanup(dev);
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);
@@ -796,16 +823,16 @@ static void __device_release_driver(stru
 	}
 }
 
-static void device_release_driver_internal(struct device *dev,
-					   struct device_driver *drv,
-					   struct device *parent)
+void device_release_driver_internal(struct device *dev,
+				    struct device_driver *drv,
+				    struct device *parent)
 {
 	if (parent)
 		device_lock(parent);
 
 	device_lock(dev);
 	if (!drv || drv == dev->driver)
-		__device_release_driver(dev);
+		__device_release_driver(dev, parent);
 
 	device_unlock(dev);
 	if (parent)
@@ -818,6 +845,10 @@ static void device_release_driver_intern
  *
  * Manually detach device from driver.
  * When called for a USB interface, @dev->parent lock must be held.
+ *
+ * If this function is to be called with @dev->parent lock held, ensure that
+ * the device's consumers are unbound in advance or that their locks can be
+ * acquired under the @dev->parent lock.
  */
 void device_release_driver(struct device *dev)
 {
Index: linux-pm/include/linux/device.h
===================================================================
--- linux-pm.orig/include/linux/device.h
+++ linux-pm/include/linux/device.h
@@ -707,6 +707,79 @@ struct device_dma_parameters {
 };
 
 /**
+ * enum device_link_state - Device link states.
+ * @NONE: The presence of the drivers is not being tracked.
+ * @DORMANT: None of the supplier/consumer drivers is present.
+ * @AVAILABLE: The supplier driver is present, but the consumer is not.
+ * @CONSUMER_PROBE: The supplier driver is present and the consumer is probing.
+ * @ACTIVE: Active link; both the supplier and consumer drivers are present.
+ * @SUPPLIER_UNBIND: The supplier driver is unbinding.
+ */
+enum device_link_state {
+	DL_STATE_NONE = -1,
+	DL_STATE_DORMANT = 0,
+	DL_STATE_AVAILABLE,
+	DL_STATE_CONSUMER_PROBE,
+	DL_STATE_ACTIVE,
+	DL_STATE_SUPPLIER_UNBIND,
+};
+
+/*
+ * Device link flags.
+ *
+ * STATELESS: The core won't track the presence of supplier/consumer drivers.
+ * AUTOREMOVE: Remove this link automatically on consumer driver unbind.
+ */
+#define DL_FLAG_STATELESS	(1 << 0)
+#define DL_FLAG_AUTOREMOVE	(1 << 1)
+
+/**
+ * struct device_link - Device link representation.
+ * @supplier: The device on the supplier end of the link.
+ * @s_node: Hook to the supplier device's list of links to consumers.
+ * @consumer: The device on the consumer end of the link.
+ * @c_node: Hook to the consumer device's list of links to suppliers.
+ * @status: The state of the link (with respect to the presence of drivers).
+ * @flags: Link flags.
+ * @rcu_head: An RCU head to use for deferred execution of SRCU callbacks.
+ */
+struct device_link {
+	struct device *supplier;
+	struct list_head s_node;
+	struct device *consumer;
+	struct list_head c_node;
+	enum device_link_state status;
+	u32 flags;
+	struct rcu_head rcu_head;
+};
+
+/**
+ * enum dl_dev_state - Device driver presence tracking information.
+ * @NO_DRIVER: There is no driver attached to the device.
+ * @PROBING: A driver is probing.
+ * @DRIVER_BOUND: The driver has been bound to the device.
+ * @UNBINDING: The driver is unbinding from the device.
+ */
+enum dl_dev_state {
+	DL_DEV_NO_DRIVER = 0,
+	DL_DEV_PROBING,
+	DL_DEV_DRIVER_BOUND,
+	DL_DEV_UNBINDING,
+};
+
+/**
+ * struct dev_links_info - Device data related to device links.
+ * @suppliers: List of links to supplier devices.
+ * @consumers: List of links to consumer devices.
+ * @status: Driver status information.
+ */
+struct dev_links_info {
+	struct list_head suppliers;
+	struct list_head consumers;
+	enum dl_dev_state status;
+};
+
+/**
  * struct device - The basic device structure
  * @parent:	The device's "parent" device, the device to which it is attached.
  * 		In most cases, a parent device is some sort of bus or host
@@ -797,6 +870,7 @@ struct device {
 					   core doesn't touch it */
 	void		*driver_data;	/* Driver data, set and get with
 					   dev_set/get_drvdata */
+	struct dev_links_info	links;
 	struct dev_pm_info	power;
 	struct dev_pm_domain	*pm_domain;
 
@@ -1113,6 +1187,10 @@ extern void device_shutdown(void);
 /* debugging and troubleshooting/diagnostic helpers. */
 extern const char *dev_driver_string(const struct device *dev);
 
+/* Device links interface. */
+struct device_link *device_link_add(struct device *consumer,
+				    struct device *supplier, u32 flags);
+void device_link_del(struct device_link *link);
 
 #ifdef CONFIG_PRINTK
 
Index: linux-pm/drivers/base/power/main.c
===================================================================
--- linux-pm.orig/drivers/base/power/main.c
+++ linux-pm/drivers/base/power/main.c
@@ -131,6 +131,7 @@ void device_pm_add(struct device *dev)
 		dev_warn(dev, "parent %s should not be sleeping\n",
 			dev_name(dev->parent));
 	list_add_tail(&dev->power.entry, &dpm_list);
+	dev->power.in_dpm_list = true;
 	mutex_unlock(&dpm_list_mtx);
 }
 
@@ -145,6 +146,7 @@ void device_pm_remove(struct device *dev
 	complete_all(&dev->power.completion);
 	mutex_lock(&dpm_list_mtx);
 	list_del_init(&dev->power.entry);
+	dev->power.in_dpm_list = false;
 	mutex_unlock(&dpm_list_mtx);
 	device_wakeup_disable(dev);
 	pm_runtime_remove(dev);
Index: linux-pm/drivers/base/power/power.h
===================================================================
--- linux-pm.orig/drivers/base/power/power.h
+++ linux-pm/drivers/base/power/power.h
@@ -127,6 +127,11 @@ extern void device_pm_move_after(struct
 extern void device_pm_move_last(struct device *);
 extern void device_pm_check_callbacks(struct device *dev);
 
+static inline bool device_pm_initialized(struct device *dev)
+{
+	return dev->power.in_dpm_list;
+}
+
 #else /* !CONFIG_PM_SLEEP */
 
 static inline void device_pm_sleep_init(struct device *dev) {}
@@ -146,6 +151,11 @@ static inline void device_pm_move_last(s
 
 static inline void device_pm_check_callbacks(struct device *dev) {}
 
+static inline bool device_pm_initialized(struct device *dev)
+{
+	return device_is_registered(dev);
+}
+
 #endif /* !CONFIG_PM_SLEEP */
 
 static inline void device_pm_init(struct device *dev)
Index: linux-pm/include/linux/pm.h
===================================================================
--- linux-pm.orig/include/linux/pm.h
+++ linux-pm/include/linux/pm.h
@@ -559,6 +559,7 @@ struct dev_pm_info {
 	pm_message_t		power_state;
 	unsigned int		can_wakeup:1;
 	unsigned int		async_suspend:1;
+	bool			in_dpm_list:1;	/* Owned by the PM core */
 	bool			is_prepared:1;	/* Owned by the PM core */
 	bool			is_suspended:1;	/* Ditto */
 	bool			is_noirq_suspended:1;

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Lukas Wunner]

Hi Rafael,

sorry for not responding to v5 of your series earlier, just sending
this out now in the hope that it reaches you before your travels.

On Mon, Oct 10, 2016 at 02:51:04PM +0200, Rafael J. Wysocki wrote:
> - Modify device_links_check_suppliers(), device_links_driver_bound(),
>   device_links_no_driver(), device_links_driver_cleanup(), device_links_busy(),
>   and device_links_unbind_consumers() to walk link lists under device_links_lock
>   (to make the new "driver presence tracking" mechanism work reliably).

This change might increase boot time if drivers return -EPROBE_DEFER.
It can easily happen that such drivers are probed dozens of times,
and each time the lock will be acquired, blocking other drivers from
probing.  Granted, it's hard to measure if and in how far boot time
really increases, but somehow I liked the previous approach better.
The combination of a mutex with an RCU plus a spinlock seemed complicated
when I reviewed the patch, but I never said anything because I came
to the conclusion that the effort seemed worthwhile to keep up the
performance.

(BTW, kbuild test robot has complained that the usage of RCUs isn't
enclosed in #ifdef CONFIG_SRCU.  Just in case you haven't seen that.)


> - Redefine device_link_add() to take three arguments, the supplier and consumer
>   pointers and flags, and to figure out the initial link state automatically.

That's a good idea, however it seems to me that there's some redundancy
between the dl_dev_state and device_link_state:  AFAICS, at any given
moment the device_link_state can be computed from the supplier's and
consumer's dl_dev_state.

One could argue that caching the device_link_state is cheaper than
recomputing it every time it's needed, but often the dl_dev_state of
one of the two devices is known, so the link can only be in a subset
of all possible states, and instead of computing the device_link_state
it's sufficient to just look at the other device's dl_dev_state.

E.g. in device_links_check_suppliers() we have this:

+		if (link->status != DL_STATE_AVAILABLE) {
+			device_links_missing_supplier(dev);
+			ret = -EPROBE_DEFER;
+			break;
+		}

When this function is called we know that the consumer's dl_dev_state is
DL_DEV_PROBING.  Of interest is only the status of the supplier. The above
if-condition would seem to be equivalent to:

		if (link->supplier->links.status == DL_DEV_DRIVER_BOUND) {

So the device_link_state seems redundant, but if it's removed from
struct device_link, changes to dl_dev_state need to be synchronized
between supplier and consumers.  Perhaps this could be accomplished
by acquiring the device_lock for the other device.  E.g. when a
consumer wants to probe, before changing its own dl_dev_state it would
have to acquire the device_lock for all suppliers to prevent races
if one of them simultaneously decides to unbind (and changes its
dl_dev_state to DL_DEV_UNBINDING).  Hm, could this deadlock?


Also, I notice that the dl_dev_state is part of a device's "links"
structure, but being able to look up a device's driver state might be
useful in other cases, not just for device links.  Maybe it makes sense
to move the dl_dev_state one level up to struct device and name the
attribute "driver_status" or somesuch (like the existing "driver" and
"driver_data" attributes).


> +	/* Deterine the initial link state. */
                ^
                m

Thanks,

Lukas

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Greg Kroah-Hartman]

On Wed, Oct 26, 2016 at 01:19:02PM +0200, Lukas Wunner wrote:
> Hi Rafael,
> 
> sorry for not responding to v5 of your series earlier, just sending
> this out now in the hope that it reaches you before your travels.
> 
> On Mon, Oct 10, 2016 at 02:51:04PM +0200, Rafael J. Wysocki wrote:
> > - Modify device_links_check_suppliers(), device_links_driver_bound(),
> >   device_links_no_driver(), device_links_driver_cleanup(), device_links_busy(),
> >   and device_links_unbind_consumers() to walk link lists under device_links_lock
> >   (to make the new "driver presence tracking" mechanism work reliably).
> 
> This change might increase boot time if drivers return -EPROBE_DEFER.

"might"?  Please verify this before guessing....

And don't make this more complex than needed before actually determining
a real issue.

thanks,

greg k-h

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Lukas Wunner]

On Thu, Oct 27, 2016 at 05:25:51PM +0200, Greg Kroah-Hartman wrote:
> On Wed, Oct 26, 2016 at 01:19:02PM +0200, Lukas Wunner wrote:
> > On Mon, Oct 10, 2016 at 02:51:04PM +0200, Rafael J. Wysocki wrote:
> > > - Modify device_links_check_suppliers(), device_links_driver_bound(),
> > >   device_links_no_driver(), device_links_driver_cleanup(), device_links_busy(),
> > >   and device_links_unbind_consumers() to walk link lists under device_links_lock
> > >   (to make the new "driver presence tracking" mechanism work reliably).
> > 
> > This change might increase boot time if drivers return -EPROBE_DEFER.
> 
> "might"?  Please verify this before guessing....

I can't, my machine only uses device links for Thunderbolt hotplug ports,
and their driver (portdrv) doesn't use deferred probing.  I'm only aware
of a single device in my system whose driver causes others to defer
probing (apple-gmux), but they don't use device links, so the mutex is
only acquired very briefly because the supplier/consumer lists are empty,
hence the performance impact can't be measured on my system.  But the
situation may be different for Marek's or Hanjun's use cases. I'm not
familiar with their systems, hence the "might".


> And don't make this more complex than needed before actually determining
> a real issue.

As pointed out it currently *is* more complex than needed because the
device_link_state is dispensable.  The whole issue is moot with the
changes I suggested because the mutex would only have to be taken for
addition/deletion of device links, not when probing.

Thanks,

Lukas

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Luis R. Rodriguez]

On Thu, Oct 27, 2016 at 05:25:51PM +0200, Greg Kroah-Hartman wrote:
> On Wed, Oct 26, 2016 at 01:19:02PM +0200, Lukas Wunner wrote:
> > Hi Rafael,
> > 
> > sorry for not responding to v5 of your series earlier, just sending
> > this out now in the hope that it reaches you before your travels.
> > 
> > On Mon, Oct 10, 2016 at 02:51:04PM +0200, Rafael J. Wysocki wrote:
> > > - Modify device_links_check_suppliers(), device_links_driver_bound(),
> > >   device_links_no_driver(), device_links_driver_cleanup(), device_links_busy(),
> > >   and device_links_unbind_consumers() to walk link lists under device_links_lock
> > >   (to make the new "driver presence tracking" mechanism work reliably).
> > 
> > This change might increase boot time if drivers return -EPROBE_DEFER.
> 
> "might"?  Please verify this before guessing....
> 
> And don't make this more complex than needed before actually determining
> a real issue.

As clarified by Rafael at Plumbers, this functional dependencies
framework assumes your driver / subsystem supports deferred probe,
if it does not support its not clear what will happen....

We have no explicit semantics to check if a driver / subsystem
supports deferred probe.

  Luis

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Greg Kroah-Hartman]

On Mon, Nov 07, 2016 at 10:22:50PM +0100, Luis R. Rodriguez wrote:
> We have no explicit semantics to check if a driver / subsystem
> supports deferred probe.

Why would we need such a thing?

thanks,

greg k-h

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Luis R. Rodriguez]

On Tue, Nov 08, 2016 at 07:45:41AM +0100, Greg Kroah-Hartman wrote:
> On Mon, Nov 07, 2016 at 10:22:50PM +0100, Luis R. Rodriguez wrote:
> > We have no explicit semantics to check if a driver / subsystem
> > supports deferred probe.
> 
> Why would we need such a thing?

It depends on the impact of a driver/subsystem not properly supporting
deffered probe, if this is no-op then such a need is not critical but
would be good to proactively inform developers / users so they avoid 
its use, if this will cause issues its perhaps best to make this a
no-op through a check. AFAICT reviewing implications of not supporting
deferred probe on drivers/subsytsems for this framework is not clearly
spelled out, if we start considering re-using this framework for probe
ordering I'd hate to see issues come up without this corner case being
concretely considered.

Furthermore -- how does this framework compare to Andrzej's resource tracking
solution? I confess I have not had a chance yet to review yet but in light of
this question it would be good to know if Andrzej's framework also requires
deferred probe as similar concerns would exist there as well.

  Luis

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Greg Kroah-Hartman]

On Tue, Nov 08, 2016 at 08:21:04PM +0100, Luis R. Rodriguez wrote:
> On Tue, Nov 08, 2016 at 07:45:41AM +0100, Greg Kroah-Hartman wrote:
> > On Mon, Nov 07, 2016 at 10:22:50PM +0100, Luis R. Rodriguez wrote:
> > > We have no explicit semantics to check if a driver / subsystem
> > > supports deferred probe.
> > 
> > Why would we need such a thing?
> 
> It depends on the impact of a driver/subsystem not properly supporting
> deffered probe, if this is no-op then such a need is not critical but
> would be good to proactively inform developers / users so they avoid 
> its use, if this will cause issues its perhaps best to make this a
> no-op through a check. AFAICT reviewing implications of not supporting
> deferred probe on drivers/subsytsems for this framework is not clearly
> spelled out, if we start considering re-using this framework for probe
> ordering I'd hate to see issues come up without this corner case being
> concretely considered.

It should not matter to the driver core if a subsystem, or a driver,
supports or does not support deferred probe.  It's a quick and simple
solution to a complex problem that works well.  Yes, you can iterate a
lot of times, but that's fine, we have time at boot to do that (and
really, it is fast.)

> Furthermore -- how does this framework compare to Andrzej's resource tracking
> solution? I confess I have not had a chance yet to review yet but in light of
> this question it would be good to know if Andrzej's framework also requires
> deferred probe as similar concerns would exist there as well.

I have no idea what "framework" you are talking about here, do you have
a pointer to patches?

thanks,

greg k-h

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Luis R. Rodriguez]

On Tue, Nov 08, 2016 at 08:43:35PM +0100, Greg Kroah-Hartman wrote:
> On Tue, Nov 08, 2016 at 08:21:04PM +0100, Luis R. Rodriguez wrote:
> > On Tue, Nov 08, 2016 at 07:45:41AM +0100, Greg Kroah-Hartman wrote:
> > > On Mon, Nov 07, 2016 at 10:22:50PM +0100, Luis R. Rodriguez wrote:
> > > > We have no explicit semantics to check if a driver / subsystem
> > > > supports deferred probe.
> > > 
> > > Why would we need such a thing?
> > 
> > It depends on the impact of a driver/subsystem not properly supporting
> > deffered probe, if this is no-op then such a need is not critical but
> > would be good to proactively inform developers / users so they avoid 
> > its use, if this will cause issues its perhaps best to make this a
> > no-op through a check. AFAICT reviewing implications of not supporting
> > deferred probe on drivers/subsytsems for this framework is not clearly
> > spelled out, if we start considering re-using this framework for probe
> > ordering I'd hate to see issues come up without this corner case being
> > concretely considered.
> 
> It should not matter to the driver core if a subsystem, or a driver,
> supports or does not support deferred probe.

That was my impression as well -- however Rafael noted this as an area
worth highlighting. So perhaps he can elaborate.

But at least as per my notes I do have here Geert Uytterhoeven reminding
us that:

  "Some drivers / subsystems don’t support deferred probe yet, such failures
   usually don’t blow up, but cause subtle malfunctioning. Example, an Ethernet
   phy could not get its interrupt as the primary IRQ chip had not been probed
   yet, it reverted to polling though. Sub-optimal." [0]

[0] https://lists.linuxfoundation.org/pipermail/ksummit-discuss/2016-August/003425.html

This sounds more like the existing deffered probe solution has detrimental
unexpected effects on some drivers / subsystems -- its not clear *why*, but
its worth reviewing if there are other drivers and seeing if we need this
annotation.

> It's a quick and simple solution to a complex problem that works well.

We all agree. The recent discussions over probe ordering are simply
optimization considerations -- should the time incurred to use deferred
probe be X and the time incurred when using an alternative is Y and we
determine the time Y is < X we have a winning alternative. Part of my
own big issue with deferred probe is a) its extremely non-deterministic,
b) it seems some folks have thought about similar problems and we might
really be able to do better. I'm not convinced that the functional
dependencies patches are the panacea for probe ordering, and while it
was not intended for that, some folks are assuming it could be. To really
vet this prospect we must really consider what other subsystem have done
and review other alternatives efforts.

> Yes, you can iterate a
> lot of times, but that's fine, we have time at boot to do that (and
> really, it is fast.)

Deferred probe is left for late_initcall() [1] -- this *assumes* that the
driver/subsystem can be loaded so late, and as per Andrzej this solution
isunacceptable/undesirable. So it would be unfair and incorrect to categorize
all drivers in the same boat, in fact given this lone fact it may be
worth revisiting the idea I mentioned about a capability aspect to support
a late deferred probe later. We tend to assume its fine, however it does
not seem to be the case.

[1] drivers/base/dd.c:late_initcall(deferred_probe_initcall);

> > Furthermore -- how does this framework compare to Andrzej's resource tracking
> > solution? I confess I have not had a chance yet to review yet but in light of
> > this question it would be good to know if Andrzej's framework also requires
> > deferred probe as similar concerns would exist there as well.
> 
> I have no idea what "framework" you are talking about here, do you have
> a pointer to patches?

I'm surprised given Andrzej did both Cc you on his patches [2] *and* chimed
in on Rafael's patches to indicate that we likely can integrate PM concerns
into his own "framework" [3]. There was no resolution to this discussion, however
its not IMHO sufficient to brush off Andrzej's points in particular because
Andrzej *is* indicating that his framework:

- Eliminates deferred probe and resulting late_initcall(), consumer registers
callbacks informing when given resources (clock, regulator, etc) becomes
available
- Properly handle resource disappearance (driver unbind, hotplug)
- Track resources which are not vital to the device, but can influence behavior
- Offers simplified resource allocation
- Can be easily expanded to help with power management

Granted I have not reviewed this yet but it at least was on my radar, and
I do believe its worth reviewing this further given the generally expressed
interest to see if we can have a common framework to address both ordering
problems, suspend and probe. At a quick glance the "ghost provider" idea
seems like a rather crazy idea but hey, there may be some goods in there.

It was sad both Andrzej and yourself could not attend the complex dependencies
tracks -- I think it would have been useful. I don't expect us to address a
resolution to probe ordering immediately -- but I am in the hopes we at least
can keep an open mind about the similarity of the problems and see if we can
aim for a clean elegant solution that might help both.

[2] https://lwn.net/Articles/625454/
[3] http://thread.gmane.org/gmane.linux.kernel/2087152

  Luis

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Greg Kroah-Hartman]

On Tue, Nov 08, 2016 at 09:58:24PM +0100, Luis R. Rodriguez wrote:
> > > Furthermore -- how does this framework compare to Andrzej's resource tracking
> > > solution? I confess I have not had a chance yet to review yet but in light of
> > > this question it would be good to know if Andrzej's framework also requires
> > > deferred probe as similar concerns would exist there as well.
> > 
> > I have no idea what "framework" you are talking about here, do you have
> > a pointer to patches?
> 
> I'm surprised given Andrzej did both Cc you on his patches [2] *and* chimed
> in on Rafael's patches to indicate that we likely can integrate PM concerns
> into his own "framework" [3]. There was no resolution to this discussion, however
> its not IMHO sufficient to brush off Andrzej's points in particular because
> Andrzej *is* indicating that his framework:

Dude, those patches were from 2014!  I can't remember patches people
sent to me a month ago...

> - Eliminates deferred probe and resulting late_initcall(), consumer registers
> callbacks informing when given resources (clock, regulator, etc) becomes
> available
> - Properly handle resource disappearance (driver unbind, hotplug)
> - Track resources which are not vital to the device, but can influence behavior
> - Offers simplified resource allocation
> - Can be easily expanded to help with power management
> 
> Granted I have not reviewed this yet but it at least was on my radar, and
> I do believe its worth reviewing this further given the generally expressed
> interest to see if we can have a common framework to address both ordering
> problems, suspend and probe. At a quick glance the "ghost provider" idea
> seems like a rather crazy idea but hey, there may be some goods in there.

>From what I remember, and I could be totally wrong, these patches were
way too complex and required that every subsystem change their
interfaces.  That's not going to work out well, but read the email
threads for the details...

> It was sad both Andrzej and yourself could not attend the complex dependencies
> tracks -- I think it would have been useful.

Sometimes real-life gets in the way of work, sorry :(

> I don't expect us to address a
> resolution to probe ordering immediately -- but I am in the hopes we at least
> can keep an open mind about the similarity of the problems and see if we can
> aim for a clean elegant solution that might help both.

I'll always review patches of what people come up with.

thanks,

greg k-h

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Andrzej Hajda]

On 09.11.2016 07:45, Greg Kroah-Hartman wrote:
> On Tue, Nov 08, 2016 at 09:58:24PM +0100, Luis R. Rodriguez wrote:
>>>> Furthermore -- how does this framework compare to Andrzej's resource tracking
>>>> solution? I confess I have not had a chance yet to review yet but in light of
>>>> this question it would be good to know if Andrzej's framework also requires
>>>> deferred probe as similar concerns would exist there as well.
>>> I have no idea what "framework" you are talking about here, do you have
>>> a pointer to patches?
>> I'm surprised given Andrzej did both Cc you on his patches [2] *and* chimed
>> in on Rafael's patches to indicate that we likely can integrate PM concerns
>> into his own "framework" [3]. There was no resolution to this discussion, however
>> its not IMHO sufficient to brush off Andrzej's points in particular because
>> Andrzej *is* indicating that his framework:
> Dude, those patches were from 2014!  I can't remember patches people
> sent to me a month ago...
>
>> - Eliminates deferred probe and resulting late_initcall(), consumer registers
>> callbacks informing when given resources (clock, regulator, etc) becomes
>> available
>> - Properly handle resource disappearance (driver unbind, hotplug)
>> - Track resources which are not vital to the device, but can influence behavior
>> - Offers simplified resource allocation
>> - Can be easily expanded to help with power management
>>
>> Granted I have not reviewed this yet but it at least was on my radar, and
>> I do believe its worth reviewing this further given the generally expressed
>> interest to see if we can have a common framework to address both ordering
>> problems, suspend and probe. At a quick glance the "ghost provider" idea
>> seems like a rather crazy idea but hey, there may be some goods in there.
> >From what I remember, and I could be totally wrong, these patches were
> way too complex and required that every subsystem change their
> interfaces.  That's not going to work out well, but read the email
> threads for the details...

I haven't seen your comment on my patches, except few general questions
regarding one of earlier version of the framework.
So maybe you are talking about different framework.

Regarding complexity, if the subsystem have simple way of
'(un)publishing' resources it just adds single calls to restrack core:
restrack_up, restrack_down in proper places.
Additionally it adds quite simple stuff to encapsulate resource
description and allocation routines into generic *_restrack_desc
structure, see for example patch adding restrack to phy framework[1].

[1]:
https://lists.freedesktop.org/archives/dri-devel/2014-December/073759.html

Regards
Andrzej

>
>> It was sad both Andrzej and yourself could not attend the complex dependencies
>> tracks -- I think it would have been useful.
> Sometimes real-life gets in the way of work, sorry :(
>
>> I don't expect us to address a
>> resolution to probe ordering immediately -- but I am in the hopes we at least
>> can keep an open mind about the similarity of the problems and see if we can
>> aim for a clean elegant solution that might help both.
> I'll always review patches of what people come up with.
>
> thanks,
>
> greg k-h
>
>
>

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Greg Kroah-Hartman]

On Wed, Nov 09, 2016 at 10:36:54AM +0100, Andrzej Hajda wrote:
> On 09.11.2016 07:45, Greg Kroah-Hartman wrote:
> > On Tue, Nov 08, 2016 at 09:58:24PM +0100, Luis R. Rodriguez wrote:
> >>>> Furthermore -- how does this framework compare to Andrzej's resource tracking
> >>>> solution? I confess I have not had a chance yet to review yet but in light of
> >>>> this question it would be good to know if Andrzej's framework also requires
> >>>> deferred probe as similar concerns would exist there as well.
> >>> I have no idea what "framework" you are talking about here, do you have
> >>> a pointer to patches?
> >> I'm surprised given Andrzej did both Cc you on his patches [2] *and* chimed
> >> in on Rafael's patches to indicate that we likely can integrate PM concerns
> >> into his own "framework" [3]. There was no resolution to this discussion, however
> >> its not IMHO sufficient to brush off Andrzej's points in particular because
> >> Andrzej *is* indicating that his framework:
> > Dude, those patches were from 2014!  I can't remember patches people
> > sent to me a month ago...
> >
> >> - Eliminates deferred probe and resulting late_initcall(), consumer registers
> >> callbacks informing when given resources (clock, regulator, etc) becomes
> >> available
> >> - Properly handle resource disappearance (driver unbind, hotplug)
> >> - Track resources which are not vital to the device, but can influence behavior
> >> - Offers simplified resource allocation
> >> - Can be easily expanded to help with power management
> >>
> >> Granted I have not reviewed this yet but it at least was on my radar, and
> >> I do believe its worth reviewing this further given the generally expressed
> >> interest to see if we can have a common framework to address both ordering
> >> problems, suspend and probe. At a quick glance the "ghost provider" idea
> >> seems like a rather crazy idea but hey, there may be some goods in there.
> > >From what I remember, and I could be totally wrong, these patches were
> > way too complex and required that every subsystem change their
> > interfaces.  That's not going to work out well, but read the email
> > threads for the details...
> 
> I haven't seen your comment on my patches, except few general questions
> regarding one of earlier version of the framework.
> So maybe you are talking about different framework.
> 
> Regarding complexity, if the subsystem have simple way of
> '(un)publishing' resources it just adds single calls to restrack core:
> restrack_up, restrack_down in proper places.
> Additionally it adds quite simple stuff to encapsulate resource
> description and allocation routines into generic *_restrack_desc
> structure, see for example patch adding restrack to phy framework[1].

Ok, again, I have no idea what my response was to a 2 year-old patchset,
again, I can't remember my response to a patchset that was sent just a
month ago...

update it, and repost and we can all go from there if you think it is a
viable solution.

thanks,

greg k-h

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Lukas Wunner]

On Tue, Nov 08, 2016 at 09:58:24PM +0100, Luis R. Rodriguez wrote:
> On Tue, Nov 08, 2016 at 08:43:35PM +0100, Greg Kroah-Hartman wrote:
> > Yes, you can iterate a
> > lot of times, but that's fine, we have time at boot to do that (and
> > really, it is fast.)
> 
> Deferred probe is left for late_initcall() [1] -- this *assumes* that the
> driver/subsystem can be loaded so late, and as per Andrzej this solution
> isunacceptable/undesirable. So it would be unfair and incorrect to categorize
> all drivers in the same boat, in fact given this lone fact it may be
> worth revisiting the idea I mentioned about a capability aspect to support
> a late deferred probe later. We tend to assume its fine, however it does
> not seem to be the case.
> 
> [1] drivers/base/dd.c:late_initcall(deferred_probe_initcall);

Note that driver_deferred_probe_trigger() is called not only from this
late_initcall but also from driver_bound().  In other words, whenever
a driver binds successfully, deferred devices will reprobe, and devices
which had to defer will not necessarily have to wait until late_initcall
until they're reprobed.  The late_initcall is just a way to tell devices
"this is last call".

Thanks,

Lukas

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Rafael J. Wysocki]

On Mon, Nov 7, 2016 at 10:22 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> On Thu, Oct 27, 2016 at 05:25:51PM +0200, Greg Kroah-Hartman wrote:
>> On Wed, Oct 26, 2016 at 01:19:02PM +0200, Lukas Wunner wrote:
>> > Hi Rafael,
>> >
>> > sorry for not responding to v5 of your series earlier, just sending
>> > this out now in the hope that it reaches you before your travels.
>> >
>> > On Mon, Oct 10, 2016 at 02:51:04PM +0200, Rafael J. Wysocki wrote:
>> > > - Modify device_links_check_suppliers(), device_links_driver_bound(),
>> > >   device_links_no_driver(), device_links_driver_cleanup(), device_links_busy(),
>> > >   and device_links_unbind_consumers() to walk link lists under device_links_lock
>> > >   (to make the new "driver presence tracking" mechanism work reliably).
>> >
>> > This change might increase boot time if drivers return -EPROBE_DEFER.
>>
>> "might"?  Please verify this before guessing....
>>
>> And don't make this more complex than needed before actually determining
>> a real issue.
>
> As clarified by Rafael at Plumbers, this functional dependencies
> framework assumes your driver / subsystem supports deferred probe,

It isn't particularly clear what you mean by "support" here.

I guess that you mean that it will allow the ->probe callback to be
invoked for multiple times for the same device/driver combination
without issues.  If that's the case, the way the new code uses
-EPROBE_DEFER doesn't interfere with this, because it will not invoke
the ->probe callbacks for consumers at all until their (required)
suppliers are ready.

> if it does not support its not clear what will happen....

I don't see any problems here, but if you see any, please just say
what they are.

> We have no explicit semantics to check if a driver / subsystem
> supports deferred probe.

That's correct, but then do we need it?

Thanks,
Rafael

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Luis R. Rodriguez]

On Wed, Nov 9, 2016 at 4:43 PM, Rafael J. Wysocki <rafael@kernel.org> wrote:
> On Mon, Nov 7, 2016 at 10:22 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
>> On Thu, Oct 27, 2016 at 05:25:51PM +0200, Greg Kroah-Hartman wrote:
>>> On Wed, Oct 26, 2016 at 01:19:02PM +0200, Lukas Wunner wrote:
>>> > Hi Rafael,
>>> >
>>> > sorry for not responding to v5 of your series earlier, just sending
>>> > this out now in the hope that it reaches you before your travels.
>>> >
>>> > On Mon, Oct 10, 2016 at 02:51:04PM +0200, Rafael J. Wysocki wrote:
>>> > > - Modify device_links_check_suppliers(), device_links_driver_bound(),
>>> > >   device_links_no_driver(), device_links_driver_cleanup(), device_links_busy(),
>>> > >   and device_links_unbind_consumers() to walk link lists under device_links_lock
>>> > >   (to make the new "driver presence tracking" mechanism work reliably).
>>> >
>>> > This change might increase boot time if drivers return -EPROBE_DEFER.
>>>
>>> "might"?  Please verify this before guessing....
>>>
>>> And don't make this more complex than needed before actually determining
>>> a real issue.
>>
>> As clarified by Rafael at Plumbers, this functional dependencies
>> framework assumes your driver / subsystem supports deferred probe,
>
> It isn't particularly clear what you mean by "support" here.

I noted some folks had reported issues, and you acknowledged that if
deferred probe was used in some drivers and if this created an issue
the same issue would be seen with this framework. AFAICT there are two
possible issues to consider:

1) the one Geert Uytterhoeven noted. Again I'll note what he had mentioned [0].

  "Some drivers / subsystems don’t support deferred probe yet, such failures
   usually don’t blow up, but cause subtle malfunctioning. Example, an Ethernet
   phy could not get its interrupt as the primary IRQ chip had not been probed
   yet, it reverted to polling though. Sub-optimal." [0]

[0] https://lists.linuxfoundation.org/pipermail/ksummit-discuss/2016-August/003425.html

Geert can you provide more details?

2) Since deferred probe relies on late_initcall() if your driver must
load earlier than this deferred probe can create an issue. Andrzej had
you identified a driver that ran into this and had issues ? If not
this seems like a semantics thing we should consider in extending the
documentation for drivers so that driver writers are aware of this
limitation. I would suppose candidates for this would be anything not
using module_init() or late_initcall() on their inits and have a
probe.

>> if it does not support its not clear what will happen....
>
> I don't see any problems here, but if you see any, please just say
> what they are.
>
>> We have no explicit semantics to check if a driver / subsystem
>> supports deferred probe.
>
> That's correct, but then do we need it?

We can determine this by reviewing the two items above.

  Luis

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Laurent Pinchart]

Hi Luis,

On Wednesday 09 Nov 2016 16:59:30 Luis R. Rodriguez wrote:
> On Wed, Nov 9, 2016 at 4:43 PM, Rafael J. Wysocki wrote:
> > On Mon, Nov 7, 2016 at 10:22 PM, Luis R. Rodriguez wrote:
> >> On Thu, Oct 27, 2016 at 05:25:51PM +0200, Greg Kroah-Hartman wrote:
> >>> On Wed, Oct 26, 2016 at 01:19:02PM +0200, Lukas Wunner wrote:
> >>>> Hi Rafael,
> >>>> 
> >>>> sorry for not responding to v5 of your series earlier, just sending
> >>>> this out now in the hope that it reaches you before your travels.
> >>>> 
> >>>> On Mon, Oct 10, 2016 at 02:51:04PM +0200, Rafael J. Wysocki wrote:
> >>>>> - Modify device_links_check_suppliers(),
> >>>>> device_links_driver_bound(),
> >>>>> 
> >>>>>   device_links_no_driver(), device_links_driver_cleanup(),
> >>>>>   device_links_busy(), and device_links_unbind_consumers() to walk
> >>>>>   link lists under device_links_lock (to make the new "driver
> >>>>>   presence tracking" mechanism work reliably).
> >>>>
> >>>> This change might increase boot time if drivers return -EPROBE_DEFER.
> >>> 
> >>> "might"?  Please verify this before guessing....
> >>> 
> >>> And don't make this more complex than needed before actually determining
> >>> a real issue.
> >> 
> >> As clarified by Rafael at Plumbers, this functional dependencies
> >> framework assumes your driver / subsystem supports deferred probe,
> > 
> > It isn't particularly clear what you mean by "support" here.
> 
> I noted some folks had reported issues, and you acknowledged that if
> deferred probe was used in some drivers and if this created an issue
> the same issue would be seen with this framework. AFAICT there are two
> possible issues to consider:
> 
> 1) the one Geert Uytterhoeven noted. Again I'll note what he had mentioned
> [0].
> 
>   "Some drivers / subsystems don’t support deferred probe yet, such failures
> usually don’t blow up, but cause subtle malfunctioning. Example, an
> Ethernet phy could not get its interrupt as the primary IRQ chip had not
> been probed yet, it reverted to polling though. Sub-optimal." [0]
> 
> [0]
> https://lists.linuxfoundation.org/pipermail/ksummit-discuss/2016-August/003
> 425.html
> 
> Geert can you provide more details?

This is a more global issue. In many cases drivers depend on optional 
resources. They are able to operate in a degraded mode (reduced feature set, 
reduced performances, ...) when those resources are not present. They can 
easily determine at probe time whether those resources are present, but have 
no way to know, in case they're absent, whether they will be present at some 
point in the near future (due to another driver probing the device providing 
the resource for instance) or if they will never be present (for instance 
because the required driver is missing). In the first case it would make sense 
to defer probe, in the latter case deferring probe forever for missing 
optional resources would prevent the device from being probed successfully at 
all.

The functional dependencies tracking patch series isn't meant to address this 
issue. I can imagine a framework that would notify drivers of optional 
resource availability after probe time, but it would come at a high cost for 
drivers as switching between modes of operation at runtime based on the 
availability of such resources would be way more complex than a mechanism 
based on probe deferral.

> 2) Since deferred probe relies on late_initcall() if your driver must
> load earlier than this deferred probe can create an issue. Andrzej had
> you identified a driver that ran into this and had issues ? If not
> this seems like a semantics thing we should consider in extending the
> documentation for drivers so that driver writers are aware of this
> limitation. I would suppose candidates for this would be anything not
> using module_init() or late_initcall() on their inits and have a
> probe.
> 
> >> if it does not support its not clear what will happen....
> > 
> > I don't see any problems here, but if you see any, please just say
> > what they are.
> > 
> >> We have no explicit semantics to check if a driver / subsystem
> >> supports deferred probe.
> > 
> > That's correct, but then do we need it?
> 
> We can determine this by reviewing the two items above.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Luis R. Rodriguez]

On Thu, Nov 10, 2016 at 09:14:32AM +0200, Laurent Pinchart wrote:
> Hi Luis,
> 
> On Wednesday 09 Nov 2016 16:59:30 Luis R. Rodriguez wrote:
> > On Wed, Nov 9, 2016 at 4:43 PM, Rafael J. Wysocki wrote:
> > > On Mon, Nov 7, 2016 at 10:22 PM, Luis R. Rodriguez wrote:
> > >> On Thu, Oct 27, 2016 at 05:25:51PM +0200, Greg Kroah-Hartman wrote:
> > >>> On Wed, Oct 26, 2016 at 01:19:02PM +0200, Lukas Wunner wrote:
> > >>>> Hi Rafael,
> > >>>> 
> > >>>> sorry for not responding to v5 of your series earlier, just sending
> > >>>> this out now in the hope that it reaches you before your travels.
> > >>>> 
> > >>>> On Mon, Oct 10, 2016 at 02:51:04PM +0200, Rafael J. Wysocki wrote:
> > >>>>> - Modify device_links_check_suppliers(),
> > >>>>> device_links_driver_bound(),
> > >>>>> 
> > >>>>>   device_links_no_driver(), device_links_driver_cleanup(),
> > >>>>>   device_links_busy(), and device_links_unbind_consumers() to walk
> > >>>>>   link lists under device_links_lock (to make the new "driver
> > >>>>>   presence tracking" mechanism work reliably).
> > >>>>
> > >>>> This change might increase boot time if drivers return -EPROBE_DEFER.
> > >>> 
> > >>> "might"?  Please verify this before guessing....
> > >>> 
> > >>> And don't make this more complex than needed before actually determining
> > >>> a real issue.
> > >> 
> > >> As clarified by Rafael at Plumbers, this functional dependencies
> > >> framework assumes your driver / subsystem supports deferred probe,
> > > 
> > > It isn't particularly clear what you mean by "support" here.
> > 
> > I noted some folks had reported issues, and you acknowledged that if
> > deferred probe was used in some drivers and if this created an issue
> > the same issue would be seen with this framework. AFAICT there are two
> > possible issues to consider:
> > 
> > 1) the one Geert Uytterhoeven noted. Again I'll note what he had mentioned
> > [0].
> > 
> >   "Some drivers / subsystems don’t support deferred probe yet, such failures
> > usually don’t blow up, but cause subtle malfunctioning. Example, an
> > Ethernet phy could not get its interrupt as the primary IRQ chip had not
> > been probed yet, it reverted to polling though. Sub-optimal." [0]
> > 
> > [0]
> > https://lists.linuxfoundation.org/pipermail/ksummit-discuss/2016-August/003
> > 425.html
> > 
> > Geert can you provide more details?
> 
> This is a more global issue. In many cases drivers depend on optional 
> resources. They are able to operate in a degraded mode (reduced feature set, 
> reduced performances, ...) when those resources are not present. They can 
> easily determine at probe time whether those resources are present, but have 
> no way to know, in case they're absent, whether they will be present at some 
> point in the near future (due to another driver probing the device providing 
> the resource for instance) or if they will never be present (for instance 
> because the required driver is missing).

I see thanks, so -EPROBE_DEFER assumes a late_initcall() would suffice to load
all necessary requirements.

> In the first case it would make sense  to defer probe,

So if the assumption is correct then it -EPROBE_DEFER should work.

> in the latter case deferring probe forever for missing 
> optional resources would prevent the device from being probed successfully at 
> all.

Right I see. And the driver core has no way to know what things *may* be
needed.

> The functional dependencies tracking patch series isn't meant to address this 
> issue.

Right, however it does track functional dependencies for suspend/run time PM
and we were certainly in hope this could help with probe ordering *later* in
the future. This is a separate topic. But more on point, the issue here is that
this framework relies on -EPROBE_DEFER -- so the issues discussed with it still
exist and should be properly documented.

> I can imagine a framework that would notify drivers of optional 
> resource availability after probe time, but it would come at a high cost for 
> drivers as switching between modes of operation at runtime based on the 
> availability of such resources would be way more complex than a mechanism 
> based on probe deferral.

Right, I see.

This is more forward looking, but -- if we had an annotation in Kconfig/turned
to a mod info section, or to start off with just a driver MODULE_SUGGESTS() macro
to start off with it might suffice for the driver core to request_module()
annotated dependencies, such requests could be explicitly suggested as
synchronous so init + probe do run together (as-is today), after which it
could know that all possible drivers that needed to be loaded should now be
loaded. If this sounds plausible to help, do we have drivers where we can
test this on? For instance, since the functional dependency framework
annotates functional dependencies for consumers/providers for suspend/resume
and un time PM could such MODULE_SUGGESTS() annotations be considered on the
consumers to suggest the provider drivers so their own probe yields to their
providers to try first ?

  Luis

> > 2) Since deferred probe relies on late_initcall() if your driver must
> > load earlier than this deferred probe can create an issue. Andrzej had
> > you identified a driver that ran into this and had issues ? If not
> > this seems like a semantics thing we should consider in extending the
> > documentation for drivers so that driver writers are aware of this
> > limitation. I would suppose candidates for this would be anything not
> > using module_init() or late_initcall() on their inits and have a
> > probe.
> > 
> > >> if it does not support its not clear what will happen....
> > > 
> > > I don't see any problems here, but if you see any, please just say
> > > what they are.
> > > 
> > >> We have no explicit semantics to check if a driver / subsystem
> > >> supports deferred probe.
> > > 
> > > That's correct, but then do we need it?
> > 
> > We can determine this by reviewing the two items above.
> 
> -- 
> Regards,
> 
> Laurent Pinchart
> 
> 

-- 
Luis Rodriguez, SUSE LINUX GmbH
Maxfeldstrasse 5; D-90409 Nuernberg

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Greg Kroah-Hartman]

On Thu, Nov 10, 2016 at 11:04:07PM +0100, Luis R. Rodriguez wrote:
> This is more forward looking, but -- if we had an annotation in Kconfig/turned
> to a mod info section, or to start off with just a driver MODULE_SUGGESTS() macro
> to start off with it might suffice for the driver core to request_module()
> annotated dependencies, such requests could be explicitly suggested as
> synchronous so init + probe do run together (as-is today), after which it
> could know that all possible drivers that needed to be loaded should now be
> loaded. If this sounds plausible to help, do we have drivers where we can
> test this on? For instance, since the functional dependency framework
> annotates functional dependencies for consumers/providers for suspend/resume
> and un time PM could such MODULE_SUGGESTS() annotations be considered on the
> consumers to suggest the provider drivers so their own probe yields to their
> providers to try first ?

No.

Stop.

First off, the "driver core" NEVER can "know" if "all possible drivers
that should be loaded, are loaded.  That way lies madness and
impossibility.

Secondly, yet-another-section isn't going to help anything here, we
alredy "suggest" to userspace a bunch of stuff, so we get the needed
modules loaded, at sometime in the future, if they are around, and
userspace feels like it.  That's the best we can ever do.

Don't try to make this more difficult than it is please. DEFER works
today really really well, and it's really really simple.
Inter-dependancy of modules and devices connected to each other are two
different things, be careful about this.

thanks,

greg k-h

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Laurent Pinchart]

Hi Greg,

On Thursday 10 Nov 2016 23:40:54 Greg Kroah-Hartman wrote:
> On Thu, Nov 10, 2016 at 11:04:07PM +0100, Luis R. Rodriguez wrote:
> > This is more forward looking, but -- if we had an annotation in
> > Kconfig/turned to a mod info section, or to start off with just a driver
> > MODULE_SUGGESTS() macro to start off with it might suffice for the driver
> > core to request_module() annotated dependencies, such requests could be
> > explicitly suggested as synchronous so init + probe do run together
> > (as-is today), after which it could know that all possible drivers that
> > needed to be loaded should now be loaded. If this sounds plausible to
> > help, do we have drivers where we can test this on? For instance, since
> > the functional dependency framework annotates functional dependencies for
> > consumers/providers for suspend/resume and un time PM could such
> > MODULE_SUGGESTS() annotations be considered on the consumers to suggest
> > the provider drivers so their own probe yields to their providers to try
> > first ?
> 
> No.
> 
> Stop.
> 
> First off, the "driver core" NEVER can "know" if "all possible drivers
> that should be loaded, are loaded.  That way lies madness and
> impossibility.
> 
> Secondly, yet-another-section isn't going to help anything here, we
> alredy "suggest" to userspace a bunch of stuff, so we get the needed
> modules loaded, at sometime in the future, if they are around, and
> userspace feels like it.  That's the best we can ever do.
> 
> Don't try to make this more difficult than it is please. DEFER works
> today really really well, and it's really really simple.
> Inter-dependancy of modules and devices connected to each other are two
> different things, be careful about this.

One issue we don't address today is handling of optional dependencies. A 
simple example is an SPI controller that can use a DMA engine or work in PIO 
mode. At probe time the driver will request a DMA channel if the platform 
(ACPI, DT, platform data) specifies that DMA is available. This can fail for 
various reasons, one of them being that the DMA engine driver hasn't probed 
the DMA device yet. In that case the SPI controller driver will continue in 
PIO mode, ignoring the DMA engine that will later be probed. We can't defer 
probing of the SPI controller as the DMA engine driver might never get loaded, 
which would result in the SPI controller probe being deferred forever.

One solution for this type of dependency issue would be to notify the SPI 
controller driver after probe that the DMA channel is now available. I'd like 
to avoid that though, as it would drastically increase the complexity of lots 
of drivers and create lots of race conditions.

There are certain configurations that we could possibly consider as invalid. 
For instance if the SPI controller driver is built-in and the DMA engine 
driver built as a module, the user clearly shot themselves in the foot and the 
kernel can't be blamed.

For resources that can't be built as a module (IOMMUs for instance) we thus 
only have to consider the case where both drivers are built-in, as the 
resource built-in and consumer as a module should work properly from an 
ordering point of view (at least as long as we don't allow asynchronous 
probing of built-in drivers to be delayed enough for modules to be loaded...). 
In this case, if the resource driver isn't available when the consumer is 
probed, if will never be available at the consumer can safely proceed in a 
degraded mode. We would thus only need to solve the probe ordering issue.

I'm not sure how far these simple(r) solutions that consider certain cases as 
invalid would scale though, and whether we won't need a more generic solution 
at some point anyway.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Greg Kroah-Hartman]

On Fri, Nov 11, 2016 at 02:08:35AM +0200, Laurent Pinchart wrote:
> Hi Greg,
> 
> On Thursday 10 Nov 2016 23:40:54 Greg Kroah-Hartman wrote:
> > On Thu, Nov 10, 2016 at 11:04:07PM +0100, Luis R. Rodriguez wrote:
> > > This is more forward looking, but -- if we had an annotation in
> > > Kconfig/turned to a mod info section, or to start off with just a driver
> > > MODULE_SUGGESTS() macro to start off with it might suffice for the driver
> > > core to request_module() annotated dependencies, such requests could be
> > > explicitly suggested as synchronous so init + probe do run together
> > > (as-is today), after which it could know that all possible drivers that
> > > needed to be loaded should now be loaded. If this sounds plausible to
> > > help, do we have drivers where we can test this on? For instance, since
> > > the functional dependency framework annotates functional dependencies for
> > > consumers/providers for suspend/resume and un time PM could such
> > > MODULE_SUGGESTS() annotations be considered on the consumers to suggest
> > > the provider drivers so their own probe yields to their providers to try
> > > first ?
> > 
> > No.
> > 
> > Stop.
> > 
> > First off, the "driver core" NEVER can "know" if "all possible drivers
> > that should be loaded, are loaded.  That way lies madness and
> > impossibility.
> > 
> > Secondly, yet-another-section isn't going to help anything here, we
> > alredy "suggest" to userspace a bunch of stuff, so we get the needed
> > modules loaded, at sometime in the future, if they are around, and
> > userspace feels like it.  That's the best we can ever do.
> > 
> > Don't try to make this more difficult than it is please. DEFER works
> > today really really well, and it's really really simple.
> > Inter-dependancy of modules and devices connected to each other are two
> > different things, be careful about this.
> 
> One issue we don't address today is handling of optional dependencies. A 
> simple example is an SPI controller that can use a DMA engine or work in PIO 
> mode. At probe time the driver will request a DMA channel if the platform 
> (ACPI, DT, platform data) specifies that DMA is available. This can fail for 
> various reasons, one of them being that the DMA engine driver hasn't probed 
> the DMA device yet. In that case the SPI controller driver will continue in 
> PIO mode, ignoring the DMA engine that will later be probed. We can't defer 
> probing of the SPI controller as the DMA engine driver might never get loaded, 
> which would result in the SPI controller probe being deferred forever.
> 
> One solution for this type of dependency issue would be to notify the SPI 
> controller driver after probe that the DMA channel is now available. I'd like 
> to avoid that though, as it would drastically increase the complexity of lots 
> of drivers and create lots of race conditions.
> 
> There are certain configurations that we could possibly consider as invalid. 
> For instance if the SPI controller driver is built-in and the DMA engine 
> driver built as a module, the user clearly shot themselves in the foot and the 
> kernel can't be blamed.
> 
> For resources that can't be built as a module (IOMMUs for instance) we thus 
> only have to consider the case where both drivers are built-in, as the 
> resource built-in and consumer as a module should work properly from an 
> ordering point of view (at least as long as we don't allow asynchronous 
> probing of built-in drivers to be delayed enough for modules to be loaded...). 
> In this case, if the resource driver isn't available when the consumer is 
> probed, if will never be available at the consumer can safely proceed in a 
> degraded mode. We would thus only need to solve the probe ordering issue.
> 
> I'm not sure how far these simple(r) solutions that consider certain cases as 
> invalid would scale though, and whether we won't need a more generic solution 
> at some point anyway.

I would love to see a generic solution that works for all of these
complex cases, as I agree with you, it's complex :)

But I have yet to see any such patches that implement this.  As always,
I am very glad to review anything that people create, but I don't have
the time to work on such a solution myself at the moment.

thanks,

greg k-h

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Luis R. Rodriguez]

On Sun, Nov 13, 2016 at 11:59:42AM +0100, Greg Kroah-Hartman wrote:
> On Fri, Nov 11, 2016 at 02:08:35AM +0200, Laurent Pinchart wrote:
> > Hi Greg,
> > 
> > On Thursday 10 Nov 2016 23:40:54 Greg Kroah-Hartman wrote:
> > > On Thu, Nov 10, 2016 at 11:04:07PM +0100, Luis R. Rodriguez wrote:
> > > > This is more forward looking, but -- if we had an annotation in
> > > > Kconfig/turned to a mod info section, or to start off with just a driver
> > > > MODULE_SUGGESTS() macro to start off with it might suffice for the driver
> > > > core to request_module() annotated dependencies, such requests could be
> > > > explicitly suggested as synchronous so init + probe do run together
> > > > (as-is today), after which it could know that all possible drivers that
> > > > needed to be loaded should now be loaded. If this sounds plausible to
> > > > help, do we have drivers where we can test this on? For instance, since
> > > > the functional dependency framework annotates functional dependencies for
> > > > consumers/providers for suspend/resume and un time PM could such
> > > > MODULE_SUGGESTS() annotations be considered on the consumers to suggest
> > > > the provider drivers so their own probe yields to their providers to try
> > > > first ?
> > > 
> > > No.
> > > 
> > > Stop.
> > > 
> > > First off, the "driver core" NEVER can "know" if "all possible drivers
> > > that should be loaded, are loaded.  That way lies madness and
> > > impossibility.

At first I had discarded the generic driver problem as a slightly unrelated
topic however its clear now its not. In terms of functional dependencies I
agree that not providing strict order is *sometimes* desirable to help with
simplicity. The generic driver problem can be described graph-wise: on a DAG,
we're considering a topology with where nodes have optional superior
alternatives, and what you seem to be advocating is a transition to an
alternative is better and much simpler than forcing order from the start. That
can only work so long as some intermediary dependencies are (for lack of a
better term) soft-nodes -- where transitions to better alternatives are
possible and can such transitions can be handled in software. You may have
(again for lack of a better term) hard-nodes though where an entry in the DAG
is required as a hard requirement immediately prior to letting another entry
proceed. An example here is the x86 IOMMU drivers and dependent GPU DRM
drivers, currently only link order asserts proper order given we have run out
of semantics in between to ensure proper order is correct.

> > > Secondly, yet-another-section isn't going to help anything here, we
> > > alredy "suggest" to userspace a bunch of stuff, so we get the needed
> > > modules loaded, at sometime in the future, if they are around, and
> > > userspace feels like it.  That's the best we can ever do.

For some cases this is sufficient, for hard-nodes (term used above) though
if you get the incorrect order you may in the worst case oops.

> > > Don't try to make this more difficult than it is please. DEFER works
> > > today really really well, and it's really really simple.

It seems many disagree. What is clear is its simplicity outweighs the
complexity by alternatives which have been historically considered. This is
reasonable. Part of the reason probe ordering, as an optimization
consideration, came up while function dependencies for runtime PM and suspend
are being discussed is as we've determined this is a related problem and
at least for hard-nodes this is critical to resolve. For now we have enough
tools to work around problems for hard-nodes, but it would be silly for us
not to start thinking about ways to improve upon this for the future.

> > > Inter-dependancy of modules and devices connected to each other are two
> > > different things, be careful about this.

This is a *very* fair warning :)

> > One issue we don't address today is handling of optional dependencies. A 
> > simple example is an SPI controller that can use a DMA engine or work in PIO 
> > mode. At probe time the driver will request a DMA channel if the platform 
> > (ACPI, DT, platform data) specifies that DMA is available. This can fail for 
> > various reasons, one of them being that the DMA engine driver hasn't probed 
> > the DMA device yet. In that case the SPI controller driver will continue in 
> > PIO mode, ignoring the DMA engine that will later be probed. We can't defer 
> > probing of the SPI controller as the DMA engine driver might never get loaded, 
> > which would result in the SPI controller probe being deferred forever.
> > 
> > One solution for this type of dependency issue would be to notify the SPI 
> > controller driver after probe that the DMA channel is now available. I'd like 
> > to avoid that though, as it would drastically increase the complexity of lots 
> > of drivers and create lots of race conditions.
> > 
> > There are certain configurations that we could possibly consider as invalid. 
> > For instance if the SPI controller driver is built-in and the DMA engine 
> > driver built as a module, the user clearly shot themselves in the foot and the 
> > kernel can't be blamed.
> > 
> > For resources that can't be built as a module (IOMMUs for instance) we thus 
> > only have to consider the case where both drivers are built-in, as the 
> > resource built-in and consumer as a module should work properly from an 
> > ordering point of view (at least as long as we don't allow asynchronous 
> > probing of built-in drivers to be delayed enough for modules to be loaded...). 
> > In this case, if the resource driver isn't available when the consumer is 
> > probed, if will never be available at the consumer can safely proceed in a 
> > degraded mode. We would thus only need to solve the probe ordering issue.
> > 
> > I'm not sure how far these simple(r) solutions that consider certain cases as 
> > invalid would scale though, and whether we won't need a more generic solution 
> > at some point anyway.
> 
> I would love to see a generic solution that works for all of these
> complex cases, as I agree with you, it's complex :)
> 
> But I have yet to see any such patches that implement this.

The generic driver topic is related but it certainly only part of the
picture. It seems there were enough folks interested in that topic though
so perhaps patches will be eventually produced for it.

> As always,
> I am very glad to review anything that people create, but I don't have
> the time to work on such a solution myself at the moment.

Part of what we tried to discuss during the complex dependencies topics at
Plumbers was evaluating if some of the existing solutions for run time PM and
suspend could help with probe ordering, it seems we had agreement on it, what
we found though was that for many cases the use of struct device for link
association is too late. Alternatives mechanisms will be considered in the
future, and it seems that one path forward will be to consider expanding upon
this simple functional device dependency framework.

So let's see more patches!

  Luis

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Geert Uytterhoeven]

Hi Laurent,

On Fri, Nov 11, 2016 at 1:08 AM, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
> On Thursday 10 Nov 2016 23:40:54 Greg Kroah-Hartman wrote:
>> On Thu, Nov 10, 2016 at 11:04:07PM +0100, Luis R. Rodriguez wrote:
>> Don't try to make this more difficult than it is please. DEFER works
>> today really really well, and it's really really simple.
>> Inter-dependancy of modules and devices connected to each other are two
>> different things, be careful about this.
>
> One issue we don't address today is handling of optional dependencies. A
> simple example is an SPI controller that can use a DMA engine or work in PIO
> mode. At probe time the driver will request a DMA channel if the platform
> (ACPI, DT, platform data) specifies that DMA is available. This can fail for
> various reasons, one of them being that the DMA engine driver hasn't probed
> the DMA device yet. In that case the SPI controller driver will continue in
> PIO mode, ignoring the DMA engine that will later be probed. We can't defer
> probing of the SPI controller as the DMA engine driver might never get loaded,
> which would result in the SPI controller probe being deferred forever.
>
> One solution for this type of dependency issue would be to notify the SPI
> controller driver after probe that the DMA channel is now available. I'd like
> to avoid that though, as it would drastically increase the complexity of lots
> of drivers and create lots of race conditions.

Alternatively, the driver can request optional DMA resources at open time
instead of at probe time.
E.g. sh_sci does that in its uart_ops.startup() callback.

Regardless of that, DMA can fail temporarily for other reasons
(e.g. running out of channels).

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Geert Uytterhoeven]

On Thu, Nov 10, 2016 at 1:59 AM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> On Wed, Nov 9, 2016 at 4:43 PM, Rafael J. Wysocki <rafael@kernel.org> wrote:
>> On Mon, Nov 7, 2016 at 10:22 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
>>> As clarified by Rafael at Plumbers, this functional dependencies
>>> framework assumes your driver / subsystem supports deferred probe,
>>
>> It isn't particularly clear what you mean by "support" here.
>
> I noted some folks had reported issues, and you acknowledged that if
> deferred probe was used in some drivers and if this created an issue
> the same issue would be seen with this framework. AFAICT there are two
> possible issues to consider:
>
> 1) the one Geert Uytterhoeven noted. Again I'll note what he had mentioned [0].
>
>   "Some drivers / subsystems don’t support deferred probe yet, such failures
>    usually don’t blow up, but cause subtle malfunctioning. Example, an Ethernet
>    phy could not get its interrupt as the primary IRQ chip had not been probed
>    yet, it reverted to polling though. Sub-optimal." [0]
>
> [0] https://lists.linuxfoundation.org/pipermail/ksummit-discuss/2016-August/003425.html
>
> Geert can you provide more details?

Issue reported in "of_mdiobus_register_phy() and deferred probe"
(http://lkml.iu.edu/hypermail/linux/kernel/1510.2/05770.html)

Key point is:
"However, of_mdiobus_register_phy() uses irq_of_parse_and_map(), which plainly
 ignores EPROBE_DEFER, and it just continues."

At that time, the PHY driver fell back to polling, but as of commit d5c3d8465
("net: phy: Avoid polling PHY with PHY_IGNORE_INTERRUPTS") that's no longer the
case, and now the PHY fails to work completely.

Workaround is "[PATCH v2] irqchip/renesas-irqc: Postpone driver initialization"
(https://www.spinics.net/lists/netdev/msg403325.html), which seems to have
sparked some interest in fixing the issue for good ;-)

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Luis R. Rodriguez]

On Thu, Nov 10, 2016 at 09:46:42AM +0100, Geert Uytterhoeven wrote:
> On Thu, Nov 10, 2016 at 1:59 AM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> > On Wed, Nov 9, 2016 at 4:43 PM, Rafael J. Wysocki <rafael@kernel.org> wrote:
> >> On Mon, Nov 7, 2016 at 10:22 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> >>> As clarified by Rafael at Plumbers, this functional dependencies
> >>> framework assumes your driver / subsystem supports deferred probe,
> >>
> >> It isn't particularly clear what you mean by "support" here.
> >
> > I noted some folks had reported issues, and you acknowledged that if
> > deferred probe was used in some drivers and if this created an issue
> > the same issue would be seen with this framework. AFAICT there are two
> > possible issues to consider:
> >
> > 1) the one Geert Uytterhoeven noted. Again I'll note what he had mentioned [0].
> >
> >   "Some drivers / subsystems don’t support deferred probe yet, such failures
> >    usually don’t blow up, but cause subtle malfunctioning. Example, an Ethernet
> >    phy could not get its interrupt as the primary IRQ chip had not been probed
> >    yet, it reverted to polling though. Sub-optimal." [0]
> >
> > [0] https://lists.linuxfoundation.org/pipermail/ksummit-discuss/2016-August/003425.html
> >
> > Geert can you provide more details?
> 
> Issue reported in "of_mdiobus_register_phy() and deferred probe"
> (http://lkml.iu.edu/hypermail/linux/kernel/1510.2/05770.html)
> 
> Key point is:
> "However, of_mdiobus_register_phy() uses irq_of_parse_and_map(), which plainly
>  ignores EPROBE_DEFER, and it just continues."
> 
> At that time, the PHY driver fell back to polling, but as of commit d5c3d8465
> ("net: phy: Avoid polling PHY with PHY_IGNORE_INTERRUPTS") that's no longer the
> case, and now the PHY fails to work completely.
> 
> Workaround is "[PATCH v2] irqchip/renesas-irqc: Postpone driver initialization"
> (https://www.spinics.net/lists/netdev/msg403325.html), which seems to have
> sparked some interest in fixing the issue for good ;-)

Ah playing with init levels. You are lucky here that this suffices, the
IOMMU folks already ran out with enough levels to play with so they cannot
resolve their issue this way, for instance.

 Luis

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Greg Kroah-Hartman]

On Mon, Oct 10, 2016 at 02:51:04PM +0200, Rafael J. Wysocki wrote:
> +/*
> + * Device link flags.
> + *
> + * STATELESS: The core won't track the presence of supplier/consumer drivers.
> + * AUTOREMOVE: Remove this link automatically on consumer driver unbind.
> + */
> +#define DL_FLAG_STATELESS	(1 << 0)
> +#define DL_FLAG_AUTOREMOVE	(1 << 1)

Minor nit, BIT()?  I'll leave this for someone to send a checkpatch
cleanup patch later on :)

thanks,

greg k-h

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Luis R. Rodriguez]

On Mon, Oct 10, 2016 at 02:51:04PM +0200, Rafael J. Wysocki wrote:
> From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> 
> Currently, there is a problem with taking functional dependencies
> between devices into account.
> 
> What I mean by a "functional dependency" is when the driver of device
> B needs device A to be functional and (generally) its driver to be
> present in order to work properly.  This has certain consequences
> for power management (suspend/resume and runtime PM ordering) and
> shutdown ordering of these devices.  In general, it also implies that
> the driver of A needs to be working for B to be probed successfully
> and it cannot be unbound from the device before the B's driver.
> 
> Support for representing those functional dependencies between
> devices is added here to allow the driver core to track them and act
> on them in certain cases where applicable.
> 
> The argument for doing that in the driver core is that there are
> quite a few distinct use cases involving device dependencies, they
> are relatively hard to get right in a driver (if one wants to
> address all of them properly) and it only gets worse if multiplied
> by the number of drivers potentially needing to do it.

How many drivers actually *need this* today for suspend / resume ?

How many of these are because of ACPI firmware bugs rather than
some other reason ?

Can ACPI somehow be used instead by these devices to address quirks?

> Morever, at
> least one case (asynchronous system suspend/resume) cannot be handled
> in a single driver at all, because it requires the driver of A to
> wait for B to suspend (during system suspend) and the driver of B to
> wait for A to resume (during system resume).

Why is this all of a sudden a new issue? It seems there is quite a bit of
frameworks already out there that somehow deal with some sort of device
ordering / dependencies, and I am curious if they've been addressing some of
these problems for a while now on their own somehow with their own solutions,
is that the case? For instance can DAPM the / DRM / Audio component framework
v4l async solution be used or does it already address some of these concerns ?

> For this reason, represent dependencies between devices as "links",
> with the help of struct device_link objects each containing pointers
> to the "linked" devices, a list node for each of them, status
> information, flags, and an RCU head for synchronization.
> 
> Also add two new list heads, representing the lists of links to the
> devices that depend on the given one (consumers) and to the devices
> depended on by it (suppliers), and a "driver presence status" field
> (needed for figuring out initial states of device links) to struct
> device.
> 
> The entire data structure consisting of all of the lists of link
> objects for all devices is protected by a mutex (for link object
> addition/removal and for list walks during device driver probing
> and removal) and by SRCU (for list walking in other case that will
> be introduced by subsequent change sets).  In addition, each link
> object has an internal status field whose value reflects whether or
> not drivers are bound to the devices pointed to by the link or
> probing/removal of their drivers is in progress etc.  That field
> is only modified under the device links mutex, but it may be read
> outside of it in some cases (introduced by subsequent change sets),
> so modifications of it are annotated with WRITE_ONCE().
> 
> New links are added by calling device_link_add() which takes three
> arguments: pointers to the devices in question and flags.  In
> particular, if DL_FLAG_STATELESS is set in the flags, the link status
> is not to be taken into account for this link and the driver core
> will not manage it.  In turn, if DL_FLAG_AUTOREMOVE is set in the
> flags, the driver core will remove the link automatically when the
> consumer device driver unbinds from it.
> 
> One of the actions carried out by device_link_add() is to reorder
> the lists used for device shutdown and system suspend/resume to
> put the consumer device along with all of its children and all of
> its consumers (and so on, recursively) to the ends of those lists
> in order to ensure the right ordering between all of the supplier
> and consumer devices.

There's no explanation as to why this order is ensured to be
correct, I think its important to document this. From our discussions
at Plumbers it seems the order is ensured due to the fact that order
was already implicitly provided through platform firmware (ACPI
enumeration is one), adjusting order on the dpm list is just shuffling
order between consumer / provider, but nothing else. In theory it
works because in the simple case this should suffice however I
remain unconvinced that if we have more users of this framework this
simple algorithm will suffice. Having a test driver or series of
test drivers that shows this would be good. As it stands there is simply
an assumption that this is correct, how *strongly* do you feel that
the order will *always* be correct if this is done and no cycles
can be added, even if tons of drivers start using this ?

> For this reason, it is not possible to create a link between two
> devices if the would-be supplier device already depends on the
> would-be consumer device as either a direct descendant of it or a
> consumer of one of its direct descendants or one of its consumers
> and so on.
> 
> There are two types of link objects, persistent and non-persistent.
> The persistent ones stay around until one of the target devices is
> deleted, while the non-persistent ones are removed automatically when
> the consumer driver unbinds from its device (ie. they are assumed to
> be valid only as long as the consumer device has a driver bound to
> it).  Persistent links are created by default and non-persistent
> links are created when the DL_FLAG_AUTOREMOVE flag is passed
> to device_link_add().
> 
> Both persistent and non-persistent device links can be deleted
> with an explicit call to device_link_del().
> 
> Links created without the DL_FLAG_STATELESS flag set are managed
> by the driver core using a simple state machine.  There are 5 states
> each link can be in: DORMANT (unused), AVAILABLE (the supplier driver
> is present and functional), CONSUMER_PROBE (the consumer driver is
> probing), ACTIVE (both supplier and consumer drivers are present and
> functional), and SUPPLIER_UNBIND (the supplier driver is unbinding).
> The driver core updates the link state automatically depending on
> what happens to the linked devices and for each link state specific
> actions are taken in addition to that.
> 
> For example, if the supplier driver unbinds from its device, the
> driver core will also unbind the drivers of all of its consumers
> automatically under the assumption that they cannot function
> properly without the supplier.  Analogously, the driver core will
> only allow the consumer driver to bind to its device if the
> supplier driver is present and functional (ie. the link is in
> the AVAILABLE state).  If that's not the case, it will rely on
> the existing deferred probing mechanism to wait for the supplier
> driver to become available.

This assumes drivers and subsystems support deferred probe, is
there a way to ensure that drivers that use this framework support
it instead of possibly breaking them if they use this framework ?

  Luis

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Rafael J. Wysocki]

On Mon, Nov 7, 2016 at 10:39 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> On Mon, Oct 10, 2016 at 02:51:04PM +0200, Rafael J. Wysocki wrote:
>> From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
>>
>> Currently, there is a problem with taking functional dependencies
>> between devices into account.
>>
>> What I mean by a "functional dependency" is when the driver of device
>> B needs device A to be functional and (generally) its driver to be
>> present in order to work properly.  This has certain consequences
>> for power management (suspend/resume and runtime PM ordering) and
>> shutdown ordering of these devices.  In general, it also implies that
>> the driver of A needs to be working for B to be probed successfully
>> and it cannot be unbound from the device before the B's driver.
>>
>> Support for representing those functional dependencies between
>> devices is added here to allow the driver core to track them and act
>> on them in certain cases where applicable.
>>
>> The argument for doing that in the driver core is that there are
>> quite a few distinct use cases involving device dependencies, they
>> are relatively hard to get right in a driver (if one wants to
>> address all of them properly) and it only gets worse if multiplied
>> by the number of drivers potentially needing to do it.
>
> How many drivers actually *need this* today for suspend / resume ?

Admittedly, I don't have a list, but from discussions I had in the
past it looked like at least a dozen.

Plus, there is this pesky ACPI _DEP thing that in some cases is
actually correct and (before this patchest) there is no way to follow
it (a classical example here is when ACPI power management methods on
one device are implemented in terms of I2C accesses and require a
specific I2C controller to be present and functional for the PM of the
device in question to work, but there are other cases like that).

> How many of these are because of ACPI firmware bugs rather than
> some other reason ?

Two things here (as I said elsewhere).  There is the order in which
operations (eg. async suspend resume of devices during system-wide
state transitions) are started and there is the order in which they
are run/completed.  Even if the former is right, the latter still may
not be correct and there needs to be a generic way to make that
happen.

The "firmware bugs" above affect the first item and even if it is
entirely correct (ie. all PM operations for all devices are always
started in the right order, consumers before suppliers or the other
way around depending on whether this is suspend or resume), there
still is the second item to address.

Apart from this, even if the information provided by the firmware does
not lead to a device registration ordering that will produce the
correct dpm_list ordering (for example), it may still not be clear
whether or not that's a bug in the firmware.  The device registration
ordering itself is just not sufficient in general.

> Can ACPI somehow be used instead by these devices to address quirks?
>
>> Morever, at
>> least one case (asynchronous system suspend/resume) cannot be handled
>> in a single driver at all, because it requires the driver of A to
>> wait for B to suspend (during system suspend) and the driver of B to
>> wait for A to resume (during system resume).
>
> Why is this all of a sudden a new issue? It seems there is quite a bit of
> frameworks already out there that somehow deal with some sort of device
> ordering / dependencies, and I am curious if they've been addressing some of
> these problems for a while now on their own somehow with their own solutions,
> is that the case? For instance can DAPM the / DRM / Audio component framework
> v4l async solution be used or does it already address some of these concerns ?

None of them don't address all of the problems involved in a
sufficiently generic way to my knowledge.

>> For this reason, represent dependencies between devices as "links",
>> with the help of struct device_link objects each containing pointers
>> to the "linked" devices, a list node for each of them, status
>> information, flags, and an RCU head for synchronization.
>>
>> Also add two new list heads, representing the lists of links to the
>> devices that depend on the given one (consumers) and to the devices
>> depended on by it (suppliers), and a "driver presence status" field
>> (needed for figuring out initial states of device links) to struct
>> device.
>>
>> The entire data structure consisting of all of the lists of link
>> objects for all devices is protected by a mutex (for link object
>> addition/removal and for list walks during device driver probing
>> and removal) and by SRCU (for list walking in other case that will
>> be introduced by subsequent change sets).  In addition, each link
>> object has an internal status field whose value reflects whether or
>> not drivers are bound to the devices pointed to by the link or
>> probing/removal of their drivers is in progress etc.  That field
>> is only modified under the device links mutex, but it may be read
>> outside of it in some cases (introduced by subsequent change sets),
>> so modifications of it are annotated with WRITE_ONCE().
>>
>> New links are added by calling device_link_add() which takes three
>> arguments: pointers to the devices in question and flags.  In
>> particular, if DL_FLAG_STATELESS is set in the flags, the link status
>> is not to be taken into account for this link and the driver core
>> will not manage it.  In turn, if DL_FLAG_AUTOREMOVE is set in the
>> flags, the driver core will remove the link automatically when the
>> consumer device driver unbinds from it.
>>
>> One of the actions carried out by device_link_add() is to reorder
>> the lists used for device shutdown and system suspend/resume to
>> put the consumer device along with all of its children and all of
>> its consumers (and so on, recursively) to the ends of those lists
>> in order to ensure the right ordering between all of the supplier
>> and consumer devices.
>
> There's no explanation as to why this order is ensured to be
> correct, I think its important to document this.

Agreed.  There is a documentation work in progress (started by Lukas).
We will get to that point.

> From our discussions
> at Plumbers it seems the order is ensured due to the fact that order
> was already implicitly provided through platform firmware (ACPI
> enumeration is one), adjusting order on the dpm list is just shuffling
> order between consumer / provider, but nothing else.

That basically is the case.

> In theory it
> works because in the simple case this should suffice however I
> remain unconvinced that if we have more users of this framework this
> simple algorithm will suffice. Having a test driver or series of
> test drivers that shows this would be good. As it stands there is simply
> an assumption that this is correct, how *strongly* do you feel that
> the order will *always* be correct if this is done and no cycles
> can be added, even if tons of drivers start using this ?

Cycles cannot be added because of the way the code works.  It will
just return NULL from device_link_add() if it detects a cycle (that it
cannot deal with).

>> For this reason, it is not possible to create a link between two
>> devices if the would-be supplier device already depends on the
>> would-be consumer device as either a direct descendant of it or a
>> consumer of one of its direct descendants or one of its consumers
>> and so on.
>>
>> There are two types of link objects, persistent and non-persistent.
>> The persistent ones stay around until one of the target devices is
>> deleted, while the non-persistent ones are removed automatically when
>> the consumer driver unbinds from its device (ie. they are assumed to
>> be valid only as long as the consumer device has a driver bound to
>> it).  Persistent links are created by default and non-persistent
>> links are created when the DL_FLAG_AUTOREMOVE flag is passed
>> to device_link_add().
>>
>> Both persistent and non-persistent device links can be deleted
>> with an explicit call to device_link_del().
>>
>> Links created without the DL_FLAG_STATELESS flag set are managed
>> by the driver core using a simple state machine.  There are 5 states
>> each link can be in: DORMANT (unused), AVAILABLE (the supplier driver
>> is present and functional), CONSUMER_PROBE (the consumer driver is
>> probing), ACTIVE (both supplier and consumer drivers are present and
>> functional), and SUPPLIER_UNBIND (the supplier driver is unbinding).
>> The driver core updates the link state automatically depending on
>> what happens to the linked devices and for each link state specific
>> actions are taken in addition to that.
>>
>> For example, if the supplier driver unbinds from its device, the
>> driver core will also unbind the drivers of all of its consumers
>> automatically under the assumption that they cannot function
>> properly without the supplier.  Analogously, the driver core will
>> only allow the consumer driver to bind to its device if the
>> supplier driver is present and functional (ie. the link is in
>> the AVAILABLE state).  If that's not the case, it will rely on
>> the existing deferred probing mechanism to wait for the supplier
>> driver to become available.
>
> This assumes drivers and subsystems support deferred probe,

No, I don't think so.

Where does that assumption matter, exactly?

> is there a way to ensure that drivers that use this framework support
> it instead of possibly breaking them if they use this framework ?

The whole driver/device binding tracking thing is not mandatory in the
first place.  You can just tell the framework to ignore that part and
just use the links for PM and shutdown.

Thanks,
Rafael

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Laurent Pinchart]

Hi Luis,

On Monday 07 Nov 2016 22:39:54 Luis R. Rodriguez wrote:
> On Mon, Oct 10, 2016 at 02:51:04PM +0200, Rafael J. Wysocki wrote:
> > From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> > 
> > Currently, there is a problem with taking functional dependencies
> > between devices into account.
> > 
> > What I mean by a "functional dependency" is when the driver of device
> > B needs device A to be functional and (generally) its driver to be
> > present in order to work properly.  This has certain consequences
> > for power management (suspend/resume and runtime PM ordering) and
> > shutdown ordering of these devices.  In general, it also implies that
> > the driver of A needs to be working for B to be probed successfully
> > and it cannot be unbound from the device before the B's driver.
> > 
> > Support for representing those functional dependencies between
> > devices is added here to allow the driver core to track them and act
> > on them in certain cases where applicable.
> > 
> > The argument for doing that in the driver core is that there are
> > quite a few distinct use cases involving device dependencies, they
> > are relatively hard to get right in a driver (if one wants to
> > address all of them properly) and it only gets worse if multiplied
> > by the number of drivers potentially needing to do it.
> 
> How many drivers actually *need this* today for suspend / resume ?

I don't think there's a list, but just speaking for Renesas R-Car platforms 
such a dependency exists from all DMA bus masters to the system IOMMUs (we're 
talking about dozens of devices here), as well as from the display, video 
codec and video processing IP cores to transparent memory access IP cores that 
handle burst access and compression/decompression.

> How many of these are because of ACPI firmware bugs rather than
> some other reason ?

None from the list above, even with s/ACPI/DT/.

> Can ACPI somehow be used instead by these devices to address quirks?

Certainly not on ARM embedded platforms :-)

> > Morever, at least one case (asynchronous system suspend/resume) cannot be
> > handled in a single driver at all, because it requires the driver of A to
> > wait for B to suspend (during system suspend) and the driver of B to
> > wait for A to resume (during system resume).
> 
> Why is this all of a sudden a new issue?

It's not a new issue, we've just never addressed it properly so far. Various 
workarounds existed, with various level of success (usually more for runtime 
PM than system suspend/resume).

> It seems there is quite a bit of frameworks already out there that somehow
> deal with some sort of device ordering / dependencies, and I am curious if
> they've been addressing some of these problems for a while now on their own
> somehow with their own solutions, is that the case? For instance can DAPM
> the / DRM / Audio component framework v4l async solution be used or does it
> already address some of these concerns ?
>
> > For this reason, represent dependencies between devices as "links",
> > with the help of struct device_link objects each containing pointers
> > to the "linked" devices, a list node for each of them, status
> > information, flags, and an RCU head for synchronization.
> > 
> > Also add two new list heads, representing the lists of links to the
> > devices that depend on the given one (consumers) and to the devices
> > depended on by it (suppliers), and a "driver presence status" field
> > (needed for figuring out initial states of device links) to struct
> > device.
> > 
> > The entire data structure consisting of all of the lists of link
> > objects for all devices is protected by a mutex (for link object
> > addition/removal and for list walks during device driver probing
> > and removal) and by SRCU (for list walking in other case that will
> > be introduced by subsequent change sets).  In addition, each link
> > object has an internal status field whose value reflects whether or
> > not drivers are bound to the devices pointed to by the link or
> > probing/removal of their drivers is in progress etc.  That field
> > is only modified under the device links mutex, but it may be read
> > outside of it in some cases (introduced by subsequent change sets),
> > so modifications of it are annotated with WRITE_ONCE().
> > 
> > New links are added by calling device_link_add() which takes three
> > arguments: pointers to the devices in question and flags.  In
> > particular, if DL_FLAG_STATELESS is set in the flags, the link status
> > is not to be taken into account for this link and the driver core
> > will not manage it.  In turn, if DL_FLAG_AUTOREMOVE is set in the
> > flags, the driver core will remove the link automatically when the
> > consumer device driver unbinds from it.
> > 
> > One of the actions carried out by device_link_add() is to reorder
> > the lists used for device shutdown and system suspend/resume to
> > put the consumer device along with all of its children and all of
> > its consumers (and so on, recursively) to the ends of those lists
> > in order to ensure the right ordering between all of the supplier
> > and consumer devices.
> 
> There's no explanation as to why this order is ensured to be
> correct, I think its important to document this. From our discussions
> at Plumbers it seems the order is ensured due to the fact that order
> was already implicitly provided through platform firmware (ACPI
> enumeration is one), adjusting order on the dpm list is just shuffling
> order between consumer / provider, but nothing else. In theory it
> works because in the simple case this should suffice however I
> remain unconvinced that if we have more users of this framework this
> simple algorithm will suffice. Having a test driver or series of
> test drivers that shows this would be good. As it stands there is simply
> an assumption that this is correct, how *strongly* do you feel that
> the order will *always* be correct if this is done and no cycles
> can be added, even if tons of drivers start using this ?
> 
> > For this reason, it is not possible to create a link between two
> > devices if the would-be supplier device already depends on the
> > would-be consumer device as either a direct descendant of it or a
> > consumer of one of its direct descendants or one of its consumers
> > and so on.
> > 
> > There are two types of link objects, persistent and non-persistent.
> > The persistent ones stay around until one of the target devices is
> > deleted, while the non-persistent ones are removed automatically when
> > the consumer driver unbinds from its device (ie. they are assumed to
> > be valid only as long as the consumer device has a driver bound to
> > it).  Persistent links are created by default and non-persistent
> > links are created when the DL_FLAG_AUTOREMOVE flag is passed
> > to device_link_add().
> > 
> > Both persistent and non-persistent device links can be deleted
> > with an explicit call to device_link_del().
> > 
> > Links created without the DL_FLAG_STATELESS flag set are managed
> > by the driver core using a simple state machine.  There are 5 states
> > each link can be in: DORMANT (unused), AVAILABLE (the supplier driver
> > is present and functional), CONSUMER_PROBE (the consumer driver is
> > probing), ACTIVE (both supplier and consumer drivers are present and
> > functional), and SUPPLIER_UNBIND (the supplier driver is unbinding).
> > The driver core updates the link state automatically depending on
> > what happens to the linked devices and for each link state specific
> > actions are taken in addition to that.
> > 
> > For example, if the supplier driver unbinds from its device, the
> > driver core will also unbind the drivers of all of its consumers
> > automatically under the assumption that they cannot function
> > properly without the supplier.  Analogously, the driver core will
> > only allow the consumer driver to bind to its device if the
> > supplier driver is present and functional (ie. the link is in
> > the AVAILABLE state).  If that's not the case, it will rely on
> > the existing deferred probing mechanism to wait for the supplier
> > driver to become available.
> 
> This assumes drivers and subsystems support deferred probe, is
> there a way to ensure that drivers that use this framework support
> it instead of possibly breaking them if they use this framework ?

-- 
Regards,

Laurent Pinchart

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Luis R. Rodriguez]

On Thu, Nov 10, 2016 at 09:05:30AM +0200, Laurent Pinchart wrote:
> Hi Luis,
> 
> On Monday 07 Nov 2016 22:39:54 Luis R. Rodriguez wrote:
> > On Mon, Oct 10, 2016 at 02:51:04PM +0200, Rafael J. Wysocki wrote:
> > > From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> > > 
> > > Currently, there is a problem with taking functional dependencies
> > > between devices into account.
> > > 
> > > What I mean by a "functional dependency" is when the driver of device
> > > B needs device A to be functional and (generally) its driver to be
> > > present in order to work properly.  This has certain consequences
> > > for power management (suspend/resume and runtime PM ordering) and
> > > shutdown ordering of these devices.  In general, it also implies that
> > > the driver of A needs to be working for B to be probed successfully
> > > and it cannot be unbound from the device before the B's driver.
> > > 
> > > Support for representing those functional dependencies between
> > > devices is added here to allow the driver core to track them and act
> > > on them in certain cases where applicable.
> > > 
> > > The argument for doing that in the driver core is that there are
> > > quite a few distinct use cases involving device dependencies, they
> > > are relatively hard to get right in a driver (if one wants to
> > > address all of them properly) and it only gets worse if multiplied
> > > by the number of drivers potentially needing to do it.
> > 
> > How many drivers actually *need this* today for suspend / resume ?
> 
> I don't think there's a list, but just speaking for Renesas R-Car platforms 
> such a dependency exists from all DMA bus masters to the system IOMMUs (we're 
> talking about dozens of devices here), as well as from the display, video 
> codec and video processing IP cores to transparent memory access IP cores that 
> handle burst access and compression/decompression.

This is very significant, thanks.

> > How many of these are because of ACPI firmware bugs rather than
> > some other reason ?
> 
> None from the list above, even with s/ACPI/DT/.
> 
> > Can ACPI somehow be used instead by these devices to address quirks?
> 
> Certainly not on ARM embedded platforms :-)

So this framework adds a generic way.

> > > Morever, at least one case (asynchronous system suspend/resume) cannot be
> > > handled in a single driver at all, because it requires the driver of A to
> > > wait for B to suspend (during system suspend) and the driver of B to
> > > wait for A to resume (during system resume).
> > 
> > Why is this all of a sudden a new issue?
> 
> It's not a new issue, we've just never addressed it properly so far. Various 
> workarounds existed, with various level of success (usually more for runtime 
> PM than system suspend/resume).

This helps a lot. I think there has at this point been enough effort to
try to inform as many folks who might have these workaround or their own
solution to voice their concerns, I tried to review some of the existing
solutions myself but its hard to get a quick grasp of them -- provided no one
else yells out I welcome then this change as a proper evolution.

I suppose the only last remaining concerns I had were aggregating on top of
deferred probe and the lack of a test driver to test complex topologies
using this framework to test order is never broken. For the first concern
it would seem that folks using this framework would actually test and ensure
device links works for their drivers, even if deferred probe is used. For
the later concern, a separate test driver could be added later.

  Luis

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Lukas Wunner]

On Mon, Nov 07, 2016 at 10:39:54PM +0100, Luis R. Rodriguez wrote:
> On Mon, Oct 10, 2016 at 02:51:04PM +0200, Rafael J. Wysocki wrote:
> > One of the actions carried out by device_link_add() is to reorder
> > the lists used for device shutdown and system suspend/resume to
> > put the consumer device along with all of its children and all of
> > its consumers (and so on, recursively) to the ends of those lists
> > in order to ensure the right ordering between all of the supplier
> > and consumer devices.
> 
> There's no explanation as to why this order is ensured to be
> correct, I think its important to document this. From our discussions
> at Plumbers it seems the order is ensured due to the fact that order
> was already implicitly provided through platform firmware (ACPI
> enumeration is one), adjusting order on the dpm list is just shuffling
> order between consumer / provider, but nothing else.

ACPI specifies a hierarchy and the order on the dpm_list and
devices_kset is such that children are behind their parent.

A device link specifies a dependency that exists in addition
to the hierarchy, hence consumers need to be moved behind
their supplier.  And not only the consumers themselves but
also recursively their children and consumers. Essentially
the entire subtree is moved to the back.  That happens in
device_reorder_to_tail() in patch 2.

If another device is enumerated which acts as a supplier to
an existing other supplier, that other supplier and all its
dependents are moved behind the newly enumerated device,
and so on.

That is provably correct so long as no loops are introduced
in the dependency graph.  That is checked by device_is_dependent(),
which is called from device_link_add(), and the addition of the
link is aborted if a loop is detected.

Best regards,

Lukas

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Luis R. Rodriguez]

On Sun, Nov 13, 2016 at 06:34:13PM +0100, Lukas Wunner wrote:
> On Mon, Nov 07, 2016 at 10:39:54PM +0100, Luis R. Rodriguez wrote:
> > On Mon, Oct 10, 2016 at 02:51:04PM +0200, Rafael J. Wysocki wrote:
> > > One of the actions carried out by device_link_add() is to reorder
> > > the lists used for device shutdown and system suspend/resume to
> > > put the consumer device along with all of its children and all of
> > > its consumers (and so on, recursively) to the ends of those lists
> > > in order to ensure the right ordering between all of the supplier
> > > and consumer devices.
> > 
> > There's no explanation as to why this order is ensured to be
> > correct, I think its important to document this. From our discussions
> > at Plumbers it seems the order is ensured due to the fact that order
> > was already implicitly provided through platform firmware (ACPI
> > enumeration is one), adjusting order on the dpm list is just shuffling
> > order between consumer / provider, but nothing else.
> 
> ACPI specifies a hierarchy and the order on the dpm_list and
> devices_kset is such that children are behind their parent.
> 
> A device link specifies a dependency that exists in addition
> to the hierarchy, hence consumers need to be moved behind
> their supplier.  And not only the consumers themselves but
> also recursively their children and consumers. Essentially
> the entire subtree is moved to the back.  That happens in
> device_reorder_to_tail() in patch 2.

Ah neat, I failed to notice this full subtree tree move, its
rather important.

> If another device is enumerated which acts as a supplier to
> an existing other supplier, that other supplier and all its
> dependents are moved behind the newly enumerated device,
> and so on.
> 
> That is probably correct so long as no loops are introduced
> in the dependency graph.

"Probably" is what concerns me, there is no formality about
the correctness of this.

> That is checked by device_is_dependent(),
> which is called from device_link_add(), and the addition of the
> link is aborted if a loop is detected.

And that is sufficient ?

 Luis

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Lukas Wunner]

On Mon, Nov 14, 2016 at 02:48:32PM +0100, Luis R. Rodriguez wrote:
> On Sun, Nov 13, 2016 at 06:34:13PM +0100, Lukas Wunner wrote:
> > On Mon, Nov 07, 2016 at 10:39:54PM +0100, Luis R. Rodriguez wrote:
> > > On Mon, Oct 10, 2016 at 02:51:04PM +0200, Rafael J. Wysocki wrote:
> > > > One of the actions carried out by device_link_add() is to reorder
> > > > the lists used for device shutdown and system suspend/resume to
> > > > put the consumer device along with all of its children and all of
> > > > its consumers (and so on, recursively) to the ends of those lists
> > > > in order to ensure the right ordering between all of the supplier
> > > > and consumer devices.
> > > 
> > > There's no explanation as to why this order is ensured to be
> > > correct, I think its important to document this. From our discussions
> > > at Plumbers it seems the order is ensured due to the fact that order
> > > was already implicitly provided through platform firmware (ACPI
> > > enumeration is one), adjusting order on the dpm list is just shuffling
> > > order between consumer / provider, but nothing else.
> > 
> > ACPI specifies a hierarchy and the order on the dpm_list and
> > devices_kset is such that children are behind their parent.
> > 
> > A device link specifies a dependency that exists in addition
> > to the hierarchy, hence consumers need to be moved behind
> > their supplier.  And not only the consumers themselves but
> > also recursively their children and consumers. Essentially
> > the entire subtree is moved to the back.  That happens in
> > device_reorder_to_tail() in patch 2.
> 
> Ah neat, I failed to notice this full subtree tree move, its
> rather important.
> 
> > If another device is enumerated which acts as a supplier to
> > an existing other supplier, that other supplier and all its
> > dependents are moved behind the newly enumerated device,
> > and so on.
> > 
> > That is probably correct so long as no loops are introduced
> > in the dependency graph.
> 
> "Probably" is what concerns me, there is no formality about
> the correctness of this.

It's a typo, I meant to say "provably correct". Sorry.

Quite a difference in meaning. :-)


> > That is checked by device_is_dependent(),
> > which is called from device_link_add(), and the addition of the
> > link is aborted if a loop is detected.
> 
> And that is sufficient ?

The device links turn the device tree into a directed acyclic graph.
For the dpm_list and devices_kset, that graph is flattened into a
one-dimensional form such that all ancestors and suppliers of a
device appear in front of that device in the lists.  I'm not a
graph theorist and can't provide a formal proof.  I think Rafael
is a Dr., maybe he can do it. :-)  I merely looked at this from a
practical point of view, i.e. I tried to come up with corner cases
where dependencies are added that would result in incorrect ordering,
and concluded that I couldn't find any.

Thanks,

Lukas

Re: [PATCH v5 2/5] driver core: Functional dependencies tracking support

[Luis R. Rodriguez]

On Mon, Nov 14, 2016 at 7:48 AM, Lukas Wunner <lukas@wunner.de> wrote:
> On Mon, Nov 14, 2016 at 02:48:32PM +0100, Luis R. Rodriguez wrote:
>> On Sun, Nov 13, 2016 at 06:34:13PM +0100, Lukas Wunner wrote:
>> > On Mon, Nov 07, 2016 at 10:39:54PM +0100, Luis R. Rodriguez wrote:
>> > > On Mon, Oct 10, 2016 at 02:51:04PM +0200, Rafael J. Wysocki wrote:
>> > > > One of the actions carried out by device_link_add() is to reorder
>> > > > the lists used for device shutdown and system suspend/resume to
>> > > > put the consumer device along with all of its children and all of
>> > > > its consumers (and so on, recursively) to the ends of those lists
>> > > > in order to ensure the right ordering between all of the supplier
>> > > > and consumer devices.
>> > >
>> > > There's no explanation as to why this order is ensured to be
>> > > correct, I think its important to document this. From our discussions
>> > > at Plumbers it seems the order is ensured due to the fact that order
>> > > was already implicitly provided through platform firmware (ACPI
>> > > enumeration is one), adjusting order on the dpm list is just shuffling
>> > > order between consumer / provider, but nothing else.
>> >
>> > ACPI specifies a hierarchy and the order on the dpm_list and
>> > devices_kset is such that children are behind their parent.
>> >
>> > A device link specifies a dependency that exists in addition
>> > to the hierarchy, hence consumers need to be moved behind
>> > their supplier.  And not only the consumers themselves but
>> > also recursively their children and consumers. Essentially
>> > the entire subtree is moved to the back.  That happens in
>> > device_reorder_to_tail() in patch 2.
>>
>> Ah neat, I failed to notice this full subtree tree move, its
>> rather important.
>>
>> > If another device is enumerated which acts as a supplier to
>> > an existing other supplier, that other supplier and all its
>> > dependents are moved behind the newly enumerated device,
>> > and so on.
>> >
>> > That is probably correct so long as no loops are introduced
>> > in the dependency graph.
>>
>> "Probably" is what concerns me, there is no formality about
>> the correctness of this.
>
> It's a typo, I meant to say "provably correct". Sorry.
>
> Quite a difference in meaning. :-)

No sorry my own mistake -- you had written provably but I thought you
had a typo, clearly you meant as you typed :)

If the trees are independent then yes, I can see this working.

>> > That is checked by device_is_dependent(),
>> > which is called from device_link_add(), and the addition of the
>> > link is aborted if a loop is detected.
>>
>> And that is sufficient ?
>
> The device links turn the device tree into a directed acyclic graph.
> For the dpm_list and devices_kset, that graph is flattened into a
> one-dimensional form such that all ancestors and suppliers of a
> device appear in front of that device in the lists.  I'm not a
> graph theorist and can't provide a formal proof.  I think Rafael
> is a Dr., maybe he can do it. :-)

If you traverse the DAG you can get a linear one-dimensional
representation of the dependencies -- I'm no expert on this either,
however I'm buying what you described above then, provided we do
indeed avoid cycles.

> I merely looked at this from a
> practical point of view, i.e. I tried to come up with corner cases
> where dependencies are added that would result in incorrect ordering,
> and concluded that I couldn't find any.

Fair enough, likewise. I think the takeaway from this discussion is a
test driver trying to break this might be good to have, but also
documenting how this works in practice and how we avoid the cycles is
important. This was not very clear from the patches.

 Luis

[PATCH v5 3/5] PM / sleep: Make async suspend/resume of devices use device links

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Make the device suspend/resume part of the core system
suspend/resume code use device links to ensure that supplier
and consumer devices will be suspended and resumed in the right
order in case of async suspend/resume.

The idea, roughly, is to use dpm_wait() to wait for all consumers
before a supplier device suspend and to wait for all suppliers
before a consumer device resume.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---

v4 -> v5:

- Rebase, change the symbol names in accordance with patch [2/5] etc.

- Use READ_ONCE() to annotate "unprotected" accesses to link status data.

---
 drivers/base/power/main.c |   85 ++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 79 insertions(+), 6 deletions(-)

Index: linux-pm/drivers/base/power/main.c
===================================================================
--- linux-pm.orig/drivers/base/power/main.c
+++ linux-pm/drivers/base/power/main.c
@@ -246,6 +246,62 @@ static void dpm_wait_for_children(struct
        device_for_each_child(dev, &async, dpm_wait_fn);
 }
 
+static void dpm_wait_for_suppliers(struct device *dev, bool async)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	/*
+	 * If the supplier goes away right after we've checked the link to it,
+	 * we'll wait for its completion to change the state, but that's fine,
+	 * because the only things that will block as a result are the SRCU
+	 * callbacks freeing the link objects for the links in the list we're
+	 * walking.
+	 */
+	list_for_each_entry_rcu(link, &dev->links.suppliers, c_node)
+		if (READ_ONCE(link->status) != DL_STATE_DORMANT)
+			dpm_wait(link->supplier, async);
+
+	device_links_read_unlock(idx);
+}
+
+static void dpm_wait_for_superior(struct device *dev, bool async)
+{
+	dpm_wait(dev->parent, async);
+	dpm_wait_for_suppliers(dev, async);
+}
+
+static void dpm_wait_for_consumers(struct device *dev, bool async)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	/*
+	 * The status of a device link can only be changed from "dormant" by a
+	 * probe, but that cannot happen during system suspend/resume.  In
+	 * theory it can change to "dormant" at that time, but then it is
+	 * reasonable to wait for the target device anyway (eg. if it goes
+	 * away, it's better to wait for it to go away completely and then
+	 * continue instead of trying to continue in parallel with its
+	 * unregistration).
+	 */
+	list_for_each_entry_rcu(link, &dev->links.consumers, s_node)
+		if (READ_ONCE(link->status) != DL_STATE_DORMANT)
+			dpm_wait(link->consumer, async);
+
+	device_links_read_unlock(idx);
+}
+
+static void dpm_wait_for_subordinate(struct device *dev, bool async)
+{
+	dpm_wait_for_children(dev, async);
+	dpm_wait_for_consumers(dev, async);
+}
+
 /**
  * pm_op - Return the PM operation appropriate for given PM event.
  * @ops: PM operations to choose from.
@@ -490,7 +546,7 @@ static int device_resume_noirq(struct de
 	if (!dev->power.is_noirq_suspended)
 		goto Out;
 
-	dpm_wait(dev->parent, async);
+	dpm_wait_for_superior(dev, async);
 
 	if (dev->pm_domain) {
 		info = "noirq power domain ";
@@ -620,7 +676,7 @@ static int device_resume_early(struct de
 	if (!dev->power.is_late_suspended)
 		goto Out;
 
-	dpm_wait(dev->parent, async);
+	dpm_wait_for_superior(dev, async);
 
 	if (dev->pm_domain) {
 		info = "early power domain ";
@@ -752,7 +808,7 @@ static int device_resume(struct device *
 		goto Complete;
 	}
 
-	dpm_wait(dev->parent, async);
+	dpm_wait_for_superior(dev, async);
 	dpm_watchdog_set(&wd, dev);
 	device_lock(dev);
 
@@ -1040,7 +1096,7 @@ static int __device_suspend_noirq(struct
 	if (dev->power.syscore || dev->power.direct_complete)
 		goto Complete;
 
-	dpm_wait_for_children(dev, async);
+	dpm_wait_for_subordinate(dev, async);
 
 	if (dev->pm_domain) {
 		info = "noirq power domain ";
@@ -1187,7 +1243,7 @@ static int __device_suspend_late(struct
 	if (dev->power.syscore || dev->power.direct_complete)
 		goto Complete;
 
-	dpm_wait_for_children(dev, async);
+	dpm_wait_for_subordinate(dev, async);
 
 	if (dev->pm_domain) {
 		info = "late power domain ";
@@ -1344,6 +1400,22 @@ static int legacy_suspend(struct device
 	return error;
 }
 
+static void dpm_clear_suppliers_direct_complete(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links.suppliers, c_node) {
+		spin_lock_irq(&link->supplier->power.lock);
+		link->supplier->power.direct_complete = false;
+		spin_unlock_irq(&link->supplier->power.lock);
+	}
+
+	device_links_read_unlock(idx);
+}
+
 /**
  * device_suspend - Execute "suspend" callbacks for given device.
  * @dev: Device to handle.
@@ -1360,7 +1432,7 @@ static int __device_suspend(struct devic
 	TRACE_DEVICE(dev);
 	TRACE_SUSPEND(0);
 
-	dpm_wait_for_children(dev, async);
+	dpm_wait_for_subordinate(dev, async);
 
 	if (async_error)
 		goto Complete;
@@ -1456,6 +1528,7 @@ static int __device_suspend(struct devic
 
 			spin_unlock_irq(&parent->power.lock);
 		}
+		dpm_clear_suppliers_direct_complete(dev);
 	}
 
 	device_unlock(dev);

[PATCH v5 4/5] PM / runtime: Use device links

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Modify the runtime PM framework to use device links to ensure that
supplier devices will not be suspended if any of their consumer
devices are active.

The idea is to reference count suppliers on the consumer's resume
and drop references to them on its suspend.  The information on
whether or not the supplier has been reference counted by the
consumer's (runtime) resume is stored in a new field (rpm_active)
in the link object for each link.

It may be necessary to clean up those references when the
supplier is unbinding and that's why the links whose status is
DEVICE_LINK_SUPPLIER_UNBIND are skipped by the runtime suspend
and resume code.

The above means that if the consumer device is probed in the
runtime-active state, the supplier has to be resumed and reference
counted by device_link_add() so the code works as expected on its
(runtime) suspend.  There is a new flag, DEVICE_LINK_RPM_ACTIVE,
to tell device_link_add() about that (in which case the caller
is responsible for making sure that the consumer really will
be runtime-active when runtime PM is enabled for it).

The other new link flag, DEVICE_LINK_PM_RUNTIME, tells the core
whether or not the link should be used for runtime PM at all.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---

v4 -> v5:

- Rebase, change the symbol names in accordance with patch [2/5] etc.

- Use READ_ONCE() to annotate "unprotected" accesses to link status data.

v3 -> v4:

- Add pm_runtime_get/put_suppliers() and make the core call them
  around the probing of a consumer device (in analogy with what it
  does to the device's parent).

---
 drivers/base/core.c          |   19 ++++-
 drivers/base/dd.c            |    3 
 drivers/base/power/runtime.c |  157 +++++++++++++++++++++++++++++++++++++++++--
 include/linux/device.h       |    5 +
 include/linux/pm_runtime.h   |    6 +
 5 files changed, 184 insertions(+), 6 deletions(-)

Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -117,6 +117,14 @@ static int device_reorder_to_tail(struct
  * @supplier: Supplier end of the link.
  * @flags: Link flags.
  *
+ * The caller is responsible for the proper synchronization of the link creation
+ * with runtime PM.  First, setting the DL_FLAG_PM_RUNTIME flag will cause the
+ * runtime PM framework to take the link into account.  Second, if the
+ * DL_FLAG_RPM_ACTIVE flag is set in addition to it, the supplier devices will
+ * be forced into the active metastate and reference-counted upon the creation
+ * of the link.  If DL_FLAG_PM_RUNTIME is not set, DL_FLAG_RPM_ACTIVE will be
+ * ignored.
+ *
  * If the DL_FLAG_AUTOREMOVE is set, the link will be removed automatically
  * when the consumer device driver unbinds from it.  The combination of both
  * DL_FLAG_AUTOREMOVE and DL_FLAG_STATELESS set is invalid and will cause NULL
@@ -158,10 +166,19 @@ struct device_link *device_link_add(stru
 		if (link->consumer == consumer)
 			goto out;
 
-	link = kmalloc(sizeof(*link), GFP_KERNEL);
+	link = kzalloc(sizeof(*link), GFP_KERNEL);
 	if (!link)
 		goto out;
 
+	if ((flags & DL_FLAG_PM_RUNTIME) && (flags & DL_FLAG_RPM_ACTIVE)) {
+		if (pm_runtime_get_sync(supplier) < 0) {
+			pm_runtime_put_noidle(supplier);
+			kfree(link);
+			link = NULL;
+			goto out;
+		}
+		link->rpm_active = true;
+	}
 	get_device(supplier);
 	link->supplier = supplier;
 	INIT_LIST_HEAD(&link->s_node);
Index: linux-pm/drivers/base/dd.c
===================================================================
--- linux-pm.orig/drivers/base/dd.c
+++ linux-pm/drivers/base/dd.c
@@ -498,6 +498,7 @@ int driver_probe_device(struct device_dr
 	pr_debug("bus: '%s': %s: matched device %s with driver %s\n",
 		 drv->bus->name, __func__, dev_name(dev), drv->name);
 
+	pm_runtime_get_suppliers(dev);
 	if (dev->parent)
 		pm_runtime_get_sync(dev->parent);
 
@@ -508,6 +509,7 @@ int driver_probe_device(struct device_dr
 	if (dev->parent)
 		pm_runtime_put(dev->parent);
 
+	pm_runtime_put_suppliers(dev);
 	return ret;
 }
 
@@ -791,6 +793,7 @@ static void __device_release_driver(stru
 		}
 
 		pm_runtime_get_sync(dev);
+		pm_runtime_clean_up_links(dev);
 
 		driver_sysfs_remove(dev);
 
Index: linux-pm/drivers/base/power/runtime.c
===================================================================
--- linux-pm.orig/drivers/base/power/runtime.c
+++ linux-pm/drivers/base/power/runtime.c
@@ -12,6 +12,8 @@
 #include <linux/pm_runtime.h>
 #include <linux/pm_wakeirq.h>
 #include <trace/events/rpm.h>
+
+#include "../base.h"
 #include "power.h"
 
 typedef int (*pm_callback_t)(struct device *);
@@ -258,6 +260,42 @@ static int rpm_check_suspend_allowed(str
 	return retval;
 }
 
+static int rpm_get_suppliers(struct device *dev)
+{
+	struct device_link *link;
+
+	list_for_each_entry_rcu(link, &dev->links.suppliers, c_node) {
+		int retval;
+
+		if (!(link->flags & DL_FLAG_PM_RUNTIME))
+			continue;
+
+		if (READ_ONCE(link->status) == DL_STATE_SUPPLIER_UNBIND ||
+		    link->rpm_active)
+			continue;
+
+		retval = pm_runtime_get_sync(link->supplier);
+		if (retval < 0) {
+			pm_runtime_put_noidle(link->supplier);
+			return retval;
+		}
+		link->rpm_active = true;
+	}
+	return 0;
+}
+
+static void rpm_put_suppliers(struct device *dev)
+{
+	struct device_link *link;
+
+	list_for_each_entry_rcu(link, &dev->links.suppliers, c_node)
+		if (link->rpm_active &&
+		    READ_ONCE(link->status) != DL_STATE_SUPPLIER_UNBIND) {
+			pm_runtime_put(link->supplier);
+			link->rpm_active = false;
+		}
+}
+
 /**
  * __rpm_callback - Run a given runtime PM callback for a given device.
  * @cb: Runtime PM callback to run.
@@ -266,19 +304,55 @@ static int rpm_check_suspend_allowed(str
 static int __rpm_callback(int (*cb)(struct device *), struct device *dev)
 	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
-	int retval;
+	int retval, idx;
 
-	if (dev->power.irq_safe)
+	if (dev->power.irq_safe) {
 		spin_unlock(&dev->power.lock);
-	else
+	} else {
 		spin_unlock_irq(&dev->power.lock);
 
+		/*
+		 * Resume suppliers if necessary.
+		 *
+		 * The device's runtime PM status cannot change until this
+		 * routine returns, so it is safe to read the status outside of
+		 * the lock.
+		 */
+		if (dev->power.runtime_status == RPM_RESUMING) {
+			idx = device_links_read_lock();
+
+			retval = rpm_get_suppliers(dev);
+			if (retval)
+				goto fail;
+
+			device_links_read_unlock(idx);
+		}
+	}
+
 	retval = cb(dev);
 
-	if (dev->power.irq_safe)
+	if (dev->power.irq_safe) {
 		spin_lock(&dev->power.lock);
-	else
+	} else {
+		/*
+		 * If the device is suspending and the callback has returned
+		 * success, drop the usage counters of the suppliers that have
+		 * been reference counted on its resume.
+		 *
+		 * Do that if resume fails too.
+		 */
+		if ((dev->power.runtime_status == RPM_SUSPENDING && !retval)
+		    || (dev->power.runtime_status == RPM_RESUMING && retval)) {
+			idx = device_links_read_lock();
+
+ fail:
+			rpm_put_suppliers(dev);
+
+			device_links_read_unlock(idx);
+		}
+
 		spin_lock_irq(&dev->power.lock);
+	}
 
 	return retval;
 }
@@ -1447,6 +1521,79 @@ void pm_runtime_remove(struct device *de
 }
 
 /**
+ * pm_runtime_clean_up_links - Prepare links to consumers for driver removal.
+ * @dev: Device whose driver is going to be removed.
+ *
+ * Check links from this device to any consumers and if any of them have active
+ * runtime PM references to the device, drop the usage counter of the device
+ * (once per link).
+ *
+ * Links with the DL_FLAG_STATELESS flag set are ignored.
+ *
+ * Since the device is guaranteed to be runtime-active at the point this is
+ * called, nothing else needs to be done here.
+ *
+ * Moreover, this is called after device_links_busy() has returned 'false', so
+ * the status of each link is guaranteed to be DL_STATE_SUPPLIER_UNBIND and
+ * therefore rpm_active can't be manipulated concurrently.
+ */
+void pm_runtime_clean_up_links(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links.consumers, s_node) {
+		if (link->flags & DL_FLAG_STATELESS)
+			continue;
+
+		if (link->rpm_active) {
+			pm_runtime_put_noidle(dev);
+			link->rpm_active = false;
+		}
+	}
+
+	device_links_read_unlock(idx);
+}
+
+/**
+ * pm_runtime_get_suppliers - Resume and reference-count supplier devices.
+ * @dev: Consumer device.
+ */
+void pm_runtime_get_suppliers(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links.suppliers, c_node)
+		if (link->flags & DL_FLAG_PM_RUNTIME)
+			pm_runtime_get_sync(link->supplier);
+
+	device_links_read_unlock(idx);
+}
+
+/**
+ * pm_runtime_put_suppliers - Drop references to supplier devices.
+ * @dev: Consumer device.
+ */
+void pm_runtime_put_suppliers(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links.suppliers, c_node)
+		if (link->flags & DL_FLAG_PM_RUNTIME)
+			pm_runtime_put(link->supplier);
+
+	device_links_read_unlock(idx);
+}
+
+/**
  * pm_runtime_force_suspend - Force a device into suspend state if needed.
  * @dev: Device to suspend.
  *
Index: linux-pm/include/linux/device.h
===================================================================
--- linux-pm.orig/include/linux/device.h
+++ linux-pm/include/linux/device.h
@@ -729,9 +729,13 @@ enum device_link_state {
  *
  * STATELESS: The core won't track the presence of supplier/consumer drivers.
  * AUTOREMOVE: Remove this link automatically on consumer driver unbind.
+ * PM_RUNTIME: If set, the runtime PM framework will use this link.
+ * RPM_ACTIVE: Run pm_runtime_get_sync() on the supplier during link creation.
  */
 #define DL_FLAG_STATELESS	(1 << 0)
 #define DL_FLAG_AUTOREMOVE	(1 << 1)
+#define DL_FLAG_PM_RUNTIME	(1 << 2)
+#define DL_FLAG_RPM_ACTIVE	(1 << 3)
 
 /**
  * struct device_link - Device link representation.
@@ -750,6 +754,7 @@ struct device_link {
 	struct list_head c_node;
 	enum device_link_state status;
 	u32 flags;
+	bool rpm_active;
 	struct rcu_head rcu_head;
 };
 
Index: linux-pm/include/linux/pm_runtime.h
===================================================================
--- linux-pm.orig/include/linux/pm_runtime.h
+++ linux-pm/include/linux/pm_runtime.h
@@ -55,6 +55,9 @@ extern unsigned long pm_runtime_autosusp
 extern void pm_runtime_update_max_time_suspended(struct device *dev,
 						 s64 delta_ns);
 extern void pm_runtime_set_memalloc_noio(struct device *dev, bool enable);
+extern void pm_runtime_clean_up_links(struct device *dev);
+extern void pm_runtime_get_suppliers(struct device *dev);
+extern void pm_runtime_put_suppliers(struct device *dev);
 
 static inline void pm_suspend_ignore_children(struct device *dev, bool enable)
 {
@@ -186,6 +189,9 @@ static inline unsigned long pm_runtime_a
 				struct device *dev) { return 0; }
 static inline void pm_runtime_set_memalloc_noio(struct device *dev,
 						bool enable){}
+static inline void pm_runtime_clean_up_links(struct device *dev) {}
+static inline void pm_runtime_get_suppliers(struct device *dev) {}
+static inline void pm_runtime_put_suppliers(struct device *dev) {}
 
 #endif /* !CONFIG_PM */
 

[Update][PATCH v5 4/5] PM / runtime: Use device links

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Modify the runtime PM framework to use device links to ensure that
supplier devices will not be suspended if any of their consumer
devices are active.

The idea is to reference count suppliers on the consumer's resume
and drop references to them on its suspend.  The information on
whether or not the supplier has been reference counted by the
consumer's (runtime) resume is stored in a new field (rpm_active)
in the link object for each link.

It may be necessary to clean up those references when the
supplier is unbinding and that's why the links whose status is
DEVICE_LINK_SUPPLIER_UNBIND are skipped by the runtime suspend
and resume code.

The above means that if the consumer device is probed in the
runtime-active state, the supplier has to be resumed and reference
counted by device_link_add() so the code works as expected on its
(runtime) suspend.  There is a new flag, DEVICE_LINK_RPM_ACTIVE,
to tell device_link_add() about that (in which case the caller
is responsible for making sure that the consumer really will
be runtime-active when runtime PM is enabled for it).

The other new link flag, DEVICE_LINK_PM_RUNTIME, tells the core
whether or not the link should be used for runtime PM at all.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---

As Marek noticed, it is necessary to balance the pm_runtime_put() done by
the core on the supplier if the link is added during consumer probe.

---
 drivers/base/core.c          |   27 +++++++
 drivers/base/dd.c            |    3 
 drivers/base/power/runtime.c |  157 +++++++++++++++++++++++++++++++++++++++++--
 include/linux/device.h       |    6 +
 include/linux/pm_runtime.h   |    6 +
 5 files changed, 193 insertions(+), 6 deletions(-)

Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -117,6 +117,14 @@ static int device_reorder_to_tail(struct
  * @supplier: Supplier end of the link.
  * @flags: Link flags.
  *
+ * The caller is responsible for the proper synchronization of the link creation
+ * with runtime PM.  First, setting the DL_FLAG_PM_RUNTIME flag will cause the
+ * runtime PM framework to take the link into account.  Second, if the
+ * DL_FLAG_RPM_ACTIVE flag is set in addition to it, the supplier devices will
+ * be forced into the active metastate and reference-counted upon the creation
+ * of the link.  If DL_FLAG_PM_RUNTIME is not set, DL_FLAG_RPM_ACTIVE will be
+ * ignored.
+ *
  * If the DL_FLAG_AUTOREMOVE is set, the link will be removed automatically
  * when the consumer device driver unbinds from it.  The combination of both
  * DL_FLAG_AUTOREMOVE and DL_FLAG_STATELESS set is invalid and will cause NULL
@@ -158,10 +166,19 @@ struct device_link *device_link_add(stru
 		if (link->consumer == consumer)
 			goto out;
 
-	link = kmalloc(sizeof(*link), GFP_KERNEL);
+	link = kzalloc(sizeof(*link), GFP_KERNEL);
 	if (!link)
 		goto out;
 
+	if ((flags & DL_FLAG_PM_RUNTIME) && (flags & DL_FLAG_RPM_ACTIVE)) {
+		if (pm_runtime_get_sync(supplier) < 0) {
+			pm_runtime_put_noidle(supplier);
+			kfree(link);
+			link = NULL;
+			goto out;
+		}
+		link->rpm_active = true;
+	}
 	get_device(supplier);
 	link->supplier = supplier;
 	INIT_LIST_HEAD(&link->s_node);
@@ -178,6 +195,14 @@ struct device_link *device_link_add(stru
 		case DL_DEV_DRIVER_BOUND:
 			switch (consumer->links.status) {
 			case DL_DEV_PROBING:
+				/*
+				 * Balance the decrementation of the supplier's
+				 * runtime PM usage counter after consumer probe
+				 * in driver_probe_device().
+				 */
+				if (flags & DL_FLAG_PM_RUNTIME)
+					pm_runtime_get_sync(supplier);
+
 				link->status = DL_STATE_CONSUMER_PROBE;
 				break;
 			case DL_DEV_DRIVER_BOUND:
Index: linux-pm/drivers/base/dd.c
===================================================================
--- linux-pm.orig/drivers/base/dd.c
+++ linux-pm/drivers/base/dd.c
@@ -498,6 +498,7 @@ int driver_probe_device(struct device_dr
 	pr_debug("bus: '%s': %s: matched device %s with driver %s\n",
 		 drv->bus->name, __func__, dev_name(dev), drv->name);
 
+	pm_runtime_get_suppliers(dev);
 	if (dev->parent)
 		pm_runtime_get_sync(dev->parent);
 
@@ -508,6 +509,7 @@ int driver_probe_device(struct device_dr
 	if (dev->parent)
 		pm_runtime_put(dev->parent);
 
+	pm_runtime_put_suppliers(dev);
 	return ret;
 }
 
@@ -791,6 +793,7 @@ static void __device_release_driver(stru
 		}
 
 		pm_runtime_get_sync(dev);
+		pm_runtime_clean_up_links(dev);
 
 		driver_sysfs_remove(dev);
 
Index: linux-pm/drivers/base/power/runtime.c
===================================================================
--- linux-pm.orig/drivers/base/power/runtime.c
+++ linux-pm/drivers/base/power/runtime.c
@@ -12,6 +12,8 @@
 #include <linux/pm_runtime.h>
 #include <linux/pm_wakeirq.h>
 #include <trace/events/rpm.h>
+
+#include "../base.h"
 #include "power.h"
 
 typedef int (*pm_callback_t)(struct device *);
@@ -258,6 +260,42 @@ static int rpm_check_suspend_allowed(str
 	return retval;
 }
 
+static int rpm_get_suppliers(struct device *dev)
+{
+	struct device_link *link;
+
+	list_for_each_entry_rcu(link, &dev->links.suppliers, c_node) {
+		int retval;
+
+		if (!(link->flags & DL_FLAG_PM_RUNTIME))
+			continue;
+
+		if (READ_ONCE(link->status) == DL_STATE_SUPPLIER_UNBIND ||
+		    link->rpm_active)
+			continue;
+
+		retval = pm_runtime_get_sync(link->supplier);
+		if (retval < 0) {
+			pm_runtime_put_noidle(link->supplier);
+			return retval;
+		}
+		link->rpm_active = true;
+	}
+	return 0;
+}
+
+static void rpm_put_suppliers(struct device *dev)
+{
+	struct device_link *link;
+
+	list_for_each_entry_rcu(link, &dev->links.suppliers, c_node)
+		if (link->rpm_active &&
+		    READ_ONCE(link->status) != DL_STATE_SUPPLIER_UNBIND) {
+			pm_runtime_put(link->supplier);
+			link->rpm_active = false;
+		}
+}
+
 /**
  * __rpm_callback - Run a given runtime PM callback for a given device.
  * @cb: Runtime PM callback to run.
@@ -266,19 +304,55 @@ static int rpm_check_suspend_allowed(str
 static int __rpm_callback(int (*cb)(struct device *), struct device *dev)
 	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
-	int retval;
+	int retval, idx;
 
-	if (dev->power.irq_safe)
+	if (dev->power.irq_safe) {
 		spin_unlock(&dev->power.lock);
-	else
+	} else {
 		spin_unlock_irq(&dev->power.lock);
 
+		/*
+		 * Resume suppliers if necessary.
+		 *
+		 * The device's runtime PM status cannot change until this
+		 * routine returns, so it is safe to read the status outside of
+		 * the lock.
+		 */
+		if (dev->power.runtime_status == RPM_RESUMING) {
+			idx = device_links_read_lock();
+
+			retval = rpm_get_suppliers(dev);
+			if (retval)
+				goto fail;
+
+			device_links_read_unlock(idx);
+		}
+	}
+
 	retval = cb(dev);
 
-	if (dev->power.irq_safe)
+	if (dev->power.irq_safe) {
 		spin_lock(&dev->power.lock);
-	else
+	} else {
+		/*
+		 * If the device is suspending and the callback has returned
+		 * success, drop the usage counters of the suppliers that have
+		 * been reference counted on its resume.
+		 *
+		 * Do that if resume fails too.
+		 */
+		if ((dev->power.runtime_status == RPM_SUSPENDING && !retval)
+		    || (dev->power.runtime_status == RPM_RESUMING && retval)) {
+			idx = device_links_read_lock();
+
+ fail:
+			rpm_put_suppliers(dev);
+
+			device_links_read_unlock(idx);
+		}
+
 		spin_lock_irq(&dev->power.lock);
+	}
 
 	return retval;
 }
@@ -1447,6 +1521,79 @@ void pm_runtime_remove(struct device *de
 }
 
 /**
+ * pm_runtime_clean_up_links - Prepare links to consumers for driver removal.
+ * @dev: Device whose driver is going to be removed.
+ *
+ * Check links from this device to any consumers and if any of them have active
+ * runtime PM references to the device, drop the usage counter of the device
+ * (once per link).
+ *
+ * Links with the DL_FLAG_STATELESS flag set are ignored.
+ *
+ * Since the device is guaranteed to be runtime-active at the point this is
+ * called, nothing else needs to be done here.
+ *
+ * Moreover, this is called after device_links_busy() has returned 'false', so
+ * the status of each link is guaranteed to be DL_STATE_SUPPLIER_UNBIND and
+ * therefore rpm_active can't be manipulated concurrently.
+ */
+void pm_runtime_clean_up_links(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links.consumers, s_node) {
+		if (link->flags & DL_FLAG_STATELESS)
+			continue;
+
+		if (link->rpm_active) {
+			pm_runtime_put_noidle(dev);
+			link->rpm_active = false;
+		}
+	}
+
+	device_links_read_unlock(idx);
+}
+
+/**
+ * pm_runtime_get_suppliers - Resume and reference-count supplier devices.
+ * @dev: Consumer device.
+ */
+void pm_runtime_get_suppliers(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links.suppliers, c_node)
+		if (link->flags & DL_FLAG_PM_RUNTIME)
+			pm_runtime_get_sync(link->supplier);
+
+	device_links_read_unlock(idx);
+}
+
+/**
+ * pm_runtime_put_suppliers - Drop references to supplier devices.
+ * @dev: Consumer device.
+ */
+void pm_runtime_put_suppliers(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links.suppliers, c_node)
+		if (link->flags & DL_FLAG_PM_RUNTIME)
+			pm_runtime_put(link->supplier);
+
+	device_links_read_unlock(idx);
+}
+
+/**
  * pm_runtime_force_suspend - Force a device into suspend state if needed.
  * @dev: Device to suspend.
  *
Index: linux-pm/include/linux/device.h
===================================================================
--- linux-pm.orig/include/linux/device.h
+++ linux-pm/include/linux/device.h
@@ -729,9 +729,13 @@ enum device_link_state {
  *
  * STATELESS: The core won't track the presence of supplier/consumer drivers.
  * AUTOREMOVE: Remove this link automatically on consumer driver unbind.
+ * PM_RUNTIME: If set, the runtime PM framework will use this link.
+ * RPM_ACTIVE: Run pm_runtime_get_sync() on the supplier during link creation.
  */
 #define DL_FLAG_STATELESS	(1 << 0)
 #define DL_FLAG_AUTOREMOVE	(1 << 1)
+#define DL_FLAG_PM_RUNTIME	(1 << 2)
+#define DL_FLAG_RPM_ACTIVE	(1 << 3)
 
 /**
  * struct device_link - Device link representation.
@@ -741,6 +745,7 @@ enum device_link_state {
  * @c_node: Hook to the consumer device's list of links to suppliers.
  * @status: The state of the link (with respect to the presence of drivers).
  * @flags: Link flags.
+ * @rpm_active: Whether or not the consumer device is runtime-PM-active.
  * @rcu_head: An RCU head to use for deferred execution of SRCU callbacks.
  */
 struct device_link {
@@ -750,6 +755,7 @@ struct device_link {
 	struct list_head c_node;
 	enum device_link_state status;
 	u32 flags;
+	bool rpm_active;
 	struct rcu_head rcu_head;
 };
 
Index: linux-pm/include/linux/pm_runtime.h
===================================================================
--- linux-pm.orig/include/linux/pm_runtime.h
+++ linux-pm/include/linux/pm_runtime.h
@@ -55,6 +55,9 @@ extern unsigned long pm_runtime_autosusp
 extern void pm_runtime_update_max_time_suspended(struct device *dev,
 						 s64 delta_ns);
 extern void pm_runtime_set_memalloc_noio(struct device *dev, bool enable);
+extern void pm_runtime_clean_up_links(struct device *dev);
+extern void pm_runtime_get_suppliers(struct device *dev);
+extern void pm_runtime_put_suppliers(struct device *dev);
 
 static inline void pm_suspend_ignore_children(struct device *dev, bool enable)
 {
@@ -186,6 +189,9 @@ static inline unsigned long pm_runtime_a
 				struct device *dev) { return 0; }
 static inline void pm_runtime_set_memalloc_noio(struct device *dev,
 						bool enable){}
+static inline void pm_runtime_clean_up_links(struct device *dev) {}
+static inline void pm_runtime_get_suppliers(struct device *dev) {}
+static inline void pm_runtime_put_suppliers(struct device *dev) {}
 
 #endif /* !CONFIG_PM */
 

[Rebase][PATCH v5 5/5] PM / runtime: Optimize the use of device links

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

If the device has no links to suppliers that should be used for
runtime PM (links with DEVICE_LINK_PM_RUNTIME set), there is no
reason to walk the list of suppliers for that device during
runtime suspend and resume.

Add a simple mechanism to detect that case and possibly avoid the
extra unnecessary overhead.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/base/core.c          |   20 +++++++++++++-------
 drivers/base/power/runtime.c |   23 ++++++++++++++++++++---
 include/linux/pm.h           |    1 +
 include/linux/pm_runtime.h   |    4 ++++
 4 files changed, 38 insertions(+), 10 deletions(-)

Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -170,14 +170,17 @@ struct device_link *device_link_add(stru
 	if (!link)
 		goto out;
 
-	if ((flags & DL_FLAG_PM_RUNTIME) && (flags & DL_FLAG_RPM_ACTIVE)) {
-		if (pm_runtime_get_sync(supplier) < 0) {
-			pm_runtime_put_noidle(supplier);
-			kfree(link);
-			link = NULL;
-			goto out;
+	if (flags & DL_FLAG_PM_RUNTIME) {
+		if (flags & DL_FLAG_RPM_ACTIVE) {
+			if (pm_runtime_get_sync(supplier) < 0) {
+				pm_runtime_put_noidle(supplier);
+				kfree(link);
+				link = NULL;
+				goto out;
+			}
+			link->rpm_active = true;
 		}
-		link->rpm_active = true;
+		pm_runtime_new_link(consumer);
 	}
 	get_device(supplier);
 	link->supplier = supplier;
@@ -250,6 +253,9 @@ static void __device_link_del(struct dev
 	dev_info(link->consumer, "Dropping the link to %s\n",
 		 dev_name(link->supplier));
 
+	if (link->flags & DL_FLAG_PM_RUNTIME)
+		pm_runtime_drop_link(link->consumer);
+
 	list_del_rcu(&link->s_node);
 	list_del_rcu(&link->c_node);
 	call_srcu(&device_links_srcu, &link->rcu_head, __device_link_free_srcu);
Index: linux-pm/drivers/base/power/runtime.c
===================================================================
--- linux-pm.orig/drivers/base/power/runtime.c
+++ linux-pm/drivers/base/power/runtime.c
@@ -305,6 +305,7 @@ static int __rpm_callback(int (*cb)(stru
 	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
 	int retval, idx;
+	bool use_links = dev->power.links_count > 0;
 
 	if (dev->power.irq_safe) {
 		spin_unlock(&dev->power.lock);
@@ -318,7 +319,7 @@ static int __rpm_callback(int (*cb)(stru
 		 * routine returns, so it is safe to read the status outside of
 		 * the lock.
 		 */
-		if (dev->power.runtime_status == RPM_RESUMING) {
+		if (use_links && dev->power.runtime_status == RPM_RESUMING) {
 			idx = device_links_read_lock();
 
 			retval = rpm_get_suppliers(dev);
@@ -341,8 +342,9 @@ static int __rpm_callback(int (*cb)(stru
 		 *
 		 * Do that if resume fails too.
 		 */
-		if ((dev->power.runtime_status == RPM_SUSPENDING && !retval)
-		    || (dev->power.runtime_status == RPM_RESUMING && retval)) {
+		if (use_links
+		    && ((dev->power.runtime_status == RPM_SUSPENDING && !retval)
+		    || (dev->power.runtime_status == RPM_RESUMING && retval))) {
 			idx = device_links_read_lock();
 
  fail:
@@ -1593,6 +1595,21 @@ void pm_runtime_put_suppliers(struct dev
 	device_links_read_unlock(idx);
 }
 
+void pm_runtime_new_link(struct device *dev)
+{
+	spin_lock_irq(&dev->power.lock);
+	dev->power.links_count++;
+	spin_unlock_irq(&dev->power.lock);
+}
+
+void pm_runtime_drop_link(struct device *dev)
+{
+	spin_lock_irq(&dev->power.lock);
+	WARN_ON(dev->power.links_count == 0);
+	dev->power.links_count--;
+	spin_unlock_irq(&dev->power.lock);
+}
+
 /**
  * pm_runtime_force_suspend - Force a device into suspend state if needed.
  * @dev: Device to suspend.
Index: linux-pm/include/linux/pm.h
===================================================================
--- linux-pm.orig/include/linux/pm.h
+++ linux-pm/include/linux/pm.h
@@ -597,6 +597,7 @@ struct dev_pm_info {
 	unsigned int		use_autosuspend:1;
 	unsigned int		timer_autosuspends:1;
 	unsigned int		memalloc_noio:1;
+	unsigned int		links_count;
 	enum rpm_request	request;
 	enum rpm_status		runtime_status;
 	int			runtime_error;
Index: linux-pm/include/linux/pm_runtime.h
===================================================================
--- linux-pm.orig/include/linux/pm_runtime.h
+++ linux-pm/include/linux/pm_runtime.h
@@ -58,6 +58,8 @@ extern void pm_runtime_set_memalloc_noio
 extern void pm_runtime_clean_up_links(struct device *dev);
 extern void pm_runtime_get_suppliers(struct device *dev);
 extern void pm_runtime_put_suppliers(struct device *dev);
+extern void pm_runtime_new_link(struct device *dev);
+extern void pm_runtime_drop_link(struct device *dev);
 
 static inline void pm_suspend_ignore_children(struct device *dev, bool enable)
 {
@@ -192,6 +194,8 @@ static inline void pm_runtime_set_memall
 static inline void pm_runtime_clean_up_links(struct device *dev) {}
 static inline void pm_runtime_get_suppliers(struct device *dev) {}
 static inline void pm_runtime_put_suppliers(struct device *dev) {}
+static inline void pm_runtime_new_link(struct device *dev) {}
+static inline void pm_runtime_drop_link(struct device *dev) {}
 
 #endif /* !CONFIG_PM */
 

Re: [PATCH v5 0/5] Functional dependencies between devices

[Marek Szyprowski]

Hi Rafael,


On 2016-10-10 14:36, Rafael J. Wysocki wrote:
> [...]
>
> One more update after some conversations during LinuxCon Europe.
>
> The main point was to make it possible for device_link_add() to figure out the
> initial state of the link instead of expecting the caller to provide it which
> might not be reliable enough in general.
>
> In this version device_link_add() takes three arguments, the supplier and
> consumer pointers and flags and it sets the correct initial state of the link
> automatically (unless invoked with the "stateless" flag, of course).  The cost
> is one additional field in struct device (I moved all of the links-related
> fields in struct device to a separate sub-structure while at it) to track
> the "driver presence status" of the device (to be used by device_link_add()).
>
> In addition to that, the links list walks in the core.c and dd.c code are
> under the device links mutex now, so the iternal link spinlock is not needed
> any more and I have renamed symbols to distinguish between flags, link states
> and device "driver presence statuses".
>
> More information is there in the changelogs.

Thanks for the update. This version is indeed easier to use and still 
works fine
with my Exynos IOMMU runtime pm rework. You can keep my:

Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>

I will send updated version of my patchset soon.

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [PATCH v5 0/5] Functional dependencies between devices

[Rafael J. Wysocki]

On Tue, Oct 18, 2016 at 12:46 PM, Marek Szyprowski
<m.szyprowski@samsung.com> wrote:
> Hi Rafael,
>
>
> On 2016-10-10 14:36, Rafael J. Wysocki wrote:
>>
>> [...]
>>
>> One more update after some conversations during LinuxCon Europe.
>>
>> The main point was to make it possible for device_link_add() to figure out
>> the
>> initial state of the link instead of expecting the caller to provide it
>> which
>> might not be reliable enough in general.
>>
>> In this version device_link_add() takes three arguments, the supplier and
>> consumer pointers and flags and it sets the correct initial state of the
>> link
>> automatically (unless invoked with the "stateless" flag, of course).  The
>> cost
>> is one additional field in struct device (I moved all of the links-related
>> fields in struct device to a separate sub-structure while at it) to track
>> the "driver presence status" of the device (to be used by
>> device_link_add()).
>>
>> In addition to that, the links list walks in the core.c and dd.c code are
>> under the device links mutex now, so the iternal link spinlock is not
>> needed
>> any more and I have renamed symbols to distinguish between flags, link
>> states
>> and device "driver presence statuses".
>>
>> More information is there in the changelogs.
>
>
> Thanks for the update. This version is indeed easier to use and still works
> fine
> with my Exynos IOMMU runtime pm rework. You can keep my:
>
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>
> I will send updated version of my patchset soon.

Thanks for the testing, much appreciated!

The series is in a new branch called "device-links-test" in my tree now:

git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git
device-links-test

Thanks,
Rafael

Re: [PATCH v5 0/5] Functional dependencies between devices

[Marek Szyprowski]

Hi Rafael,


On 2016-10-19 13:57, Rafael J. Wysocki wrote:
> On Tue, Oct 18, 2016 at 12:46 PM, Marek Szyprowski
> <m.szyprowski@samsung.com> wrote:
>> On 2016-10-10 14:36, Rafael J. Wysocki wrote:
>>> [...]
>>>
>>> One more update after some conversations during LinuxCon Europe.
>>>
>>> The main point was to make it possible for device_link_add() to figure out
>>> the
>>> initial state of the link instead of expecting the caller to provide it
>>> which
>>> might not be reliable enough in general.
>>>
>>> In this version device_link_add() takes three arguments, the supplier and
>>> consumer pointers and flags and it sets the correct initial state of the
>>> link
>>> automatically (unless invoked with the "stateless" flag, of course).  The
>>> cost
>>> is one additional field in struct device (I moved all of the links-related
>>> fields in struct device to a separate sub-structure while at it) to track
>>> the "driver presence status" of the device (to be used by
>>> device_link_add()).
>>>
>>> In addition to that, the links list walks in the core.c and dd.c code are
>>> under the device links mutex now, so the iternal link spinlock is not
>>> needed
>>> any more and I have renamed symbols to distinguish between flags, link
>>> states
>>> and device "driver presence statuses".
>>>
>>> More information is there in the changelogs.
>> Thanks for the update. This version is indeed easier to use and still works
>> fine
>> with my Exynos IOMMU runtime pm rework. You can keep my:
>>
>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>>
>> I will send updated version of my patchset soon.
> Thanks for the testing, much appreciated!
>
> The series is in a new branch called "device-links-test" in my tree now:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git
> device-links-test

While working on integrating IOMMU deferred probing patches I found a bug,
which has been introduced in v4 of device dependency patchset (v3 worked
fine in this area, v5 also contains this bug). The following fixup is
needed to properly create links with DL_FLAG_PM_RUNTIME flag set during
consumer device probing:

From: Marek Szyprowski <m.szyprowski@samsung.com>
Date: Thu, 20 Oct 2016 12:12:14 +0200
Subject: [PATCH] driver core: fix runtime pm state for
  DEVICE_LINK_CONSUMER_PROBE links

If link is added during consumer probe with DL_FLAG_PM_RUNTIME flag set,
the code will do additional pm_runtime_put() on the supplier after
finishing consumer probing. This will break runtime pm operation for
the supplier device. To solve this issue, enforce additional call to
pm_runtime_get_sync() on the supplier device while creating such link.

Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
---
  drivers/base/core.c | 3 +++
  1 file changed, 3 insertions(+)

diff --git a/drivers/base/core.c b/drivers/base/core.c
index 48bc5a362f7d..d4cc285a1df7 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -217,6 +217,9 @@ struct device_link *device_link_add(struct device 
*consumer,
          }
      }

+    if (flags & DL_FLAG_PM_RUNTIME && status == DL_STATE_CONSUMER_PROBE)
+        pm_runtime_get_sync(supplier);
+
      /*
       * Move the consumer and all of the devices depending on it to the end
       * of dpm_list and the devices_kset list.
-- 
1.9.1

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [PATCH v5 0/5] Functional dependencies between devices

[Rafael J. Wysocki]

Hi,

On Thu, Oct 20, 2016 at 12:21 PM, Marek Szyprowski
<m.szyprowski@samsung.com> wrote:
> Hi Rafael,
>
>
> On 2016-10-19 13:57, Rafael J. Wysocki wrote:
>>
>> On Tue, Oct 18, 2016 at 12:46 PM, Marek Szyprowski
>> <m.szyprowski@samsung.com> wrote:
>>>
>>> On 2016-10-10 14:36, Rafael J. Wysocki wrote:
>>>>
>>>> [...]
>>>>
>>>> One more update after some conversations during LinuxCon Europe.
>>>>
>>>> The main point was to make it possible for device_link_add() to figure
>>>> out
>>>> the
>>>> initial state of the link instead of expecting the caller to provide it
>>>> which
>>>> might not be reliable enough in general.
>>>>
>>>> In this version device_link_add() takes three arguments, the supplier
>>>> and
>>>> consumer pointers and flags and it sets the correct initial state of the
>>>> link
>>>> automatically (unless invoked with the "stateless" flag, of course).
>>>> The
>>>> cost
>>>> is one additional field in struct device (I moved all of the
>>>> links-related
>>>> fields in struct device to a separate sub-structure while at it) to
>>>> track
>>>> the "driver presence status" of the device (to be used by
>>>> device_link_add()).
>>>>
>>>> In addition to that, the links list walks in the core.c and dd.c code
>>>> are
>>>> under the device links mutex now, so the iternal link spinlock is not
>>>> needed
>>>> any more and I have renamed symbols to distinguish between flags, link
>>>> states
>>>> and device "driver presence statuses".
>>>>
>>>> More information is there in the changelogs.
>>>
>>> Thanks for the update. This version is indeed easier to use and still
>>> works
>>> fine
>>> with my Exynos IOMMU runtime pm rework. You can keep my:
>>>
>>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>>>
>>> I will send updated version of my patchset soon.
>>
>> Thanks for the testing, much appreciated!
>>
>> The series is in a new branch called "device-links-test" in my tree now:
>>
>> git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git
>> device-links-test
>
>
> While working on integrating IOMMU deferred probing patches I found a bug,
> which has been introduced in v4 of device dependency patchset (v3 worked
> fine in this area, v5 also contains this bug). The following fixup is
> needed to properly create links with DL_FLAG_PM_RUNTIME flag set during
> consumer device probing:
>
> From: Marek Szyprowski <m.szyprowski@samsung.com>
> Date: Thu, 20 Oct 2016 12:12:14 +0200
> Subject: [PATCH] driver core: fix runtime pm state for
>  DEVICE_LINK_CONSUMER_PROBE links
>
> If link is added during consumer probe with DL_FLAG_PM_RUNTIME flag set,
> the code will do additional pm_runtime_put() on the supplier after
> finishing consumer probing. This will break runtime pm operation for
> the supplier device. To solve this issue, enforce additional call to
> pm_runtime_get_sync() on the supplier device while creating such link.
>
> Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
> ---
>  drivers/base/core.c | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/drivers/base/core.c b/drivers/base/core.c
> index 48bc5a362f7d..d4cc285a1df7 100644
> --- a/drivers/base/core.c
> +++ b/drivers/base/core.c
> @@ -217,6 +217,9 @@ struct device_link *device_link_add(struct device
> *consumer,
>          }
>      }
>
> +    if (flags & DL_FLAG_PM_RUNTIME && status == DL_STATE_CONSUMER_PROBE)
> +        pm_runtime_get_sync(supplier);
> +

Good catch!

I'd prefer to do this slightly differently, though, so I'll send an
update of the runtime PM patch with this folded in shortly.

Thanks,
Rafael

Re: [PATCH v5 0/5] Functional dependencies between devices

[Greg Kroah-Hartman]

On Mon, Oct 10, 2016 at 02:36:31PM +0200, Rafael J. Wysocki wrote:
> Hi Everyone,
> 
> > On Thursday, September 08, 2016 11:25:44 PM Rafael J. Wysocki wrote:
> > 
> 
> [cut]
> 
> > 
> > Time for another update. :-)
> > 
> > Fewer changes this time, mostly to address issues found by Lukas and Marek.
> > 
> > The most significant one is to make device_link_add() cope with the case
> > when
> > the consumer device has not been registered yet when it is called.  The
> > supplier device still is required to be registered and the function will
> > return NULL if that is not the case.
> > 
> > Another significant change is in patch [4/5] that now makes the core apply
> > pm_runtime_get_sync()/pm_runtime_put() to supplier devices around the
> > probing of a consumer one (in analogy with the parent).
> 
> One more update after some conversations during LinuxCon Europe.
> 
> The main point was to make it possible for device_link_add() to figure out the
> initial state of the link instead of expecting the caller to provide it which
> might not be reliable enough in general.
> 
> In this version device_link_add() takes three arguments, the supplier and
> consumer pointers and flags and it sets the correct initial state of the link
> automatically (unless invoked with the "stateless" flag, of course).  The cost
> is one additional field in struct device (I moved all of the links-related
> fields in struct device to a separate sub-structure while at it) to track
> the "driver presence status" of the device (to be used by device_link_add()).
> 
> In addition to that, the links list walks in the core.c and dd.c code are
> under the device links mutex now, so the iternal link spinlock is not needed
> any more and I have renamed symbols to distinguish between flags, link states
> and device "driver presence statuses".
> 
> More information is there in the changelogs.

First off, thanks so much for doing this work, it's been needed for a
long time.  It all looks good, and I've added it to my testing tree to
give the 0-day bot some fun with it for a day or so.

thanks,

greg k-h

Re: [PATCH v5 0/5] Functional dependencies between devices

[Lukas Wunner]

On Thu, Oct 27, 2016 at 08:19:27PM +0800, Hanjun Guo wrote:
> I'm trying to using this patch set to solve the functional dependency
> between devices and irqchip, which are both ACPI platform devices.
> irqchip needs to be probed before the devices connecting to them,
> which specifically, it's the mbi-gen support I send out recently:
> 
> https://lkml.org/lkml/2016/10/25/453
> 
> But I didn't see an example to do so in this patch set, and seems that
> some extra code needs to be added for that purpose, could you give me
> some suggestions for how to do that then I can work on and test against
> your patch set?

If the consumers can detect that there's a consumer on which they depend,
you could call device_link_add() from their ->probe hook.

Generally the earliest point in time when device links can be added is
after device_initialize() has been called for the consumer and device_add()
has been called for the supplier.  (At least that's my understanding.)

HTH,

Lukas

Re: [PATCH v5 0/5] Functional dependencies between devices

[Hanjun Guo]

On 10/28/2016 05:39 PM, Lukas Wunner wrote:
> On Thu, Oct 27, 2016 at 08:19:27PM +0800, Hanjun Guo wrote:
>> I'm trying to using this patch set to solve the functional dependency
>> between devices and irqchip, which are both ACPI platform devices.
>> irqchip needs to be probed before the devices connecting to them,
>> which specifically, it's the mbi-gen support I send out recently:
>>
>> https://lkml.org/lkml/2016/10/25/453
>>
>> But I didn't see an example to do so in this patch set, and seems that
>> some extra code needs to be added for that purpose, could you give me
>> some suggestions for how to do that then I can work on and test against
>> your patch set?
>
> If the consumers can detect that there's a consumer on which they depend,
> you could call device_link_add() from their ->probe hook.
>
> Generally the earliest point in time when device links can be added is
> after device_initialize() has been called for the consumer and device_add()
> has been called for the supplier.  (At least that's my understanding.)

Thank you, currently I'm on travailing and will take a deep look to
see if it works (or adding things on top) on my case.

Thanks
Hanjun

[PATCH v6 0/5] Functional dependencies between devices

[Rafael J. Wysocki]

Hi,

Let me quote from the previous intro messages for this series first:

> > Time for another update. :-)
> > 
> > Fewer changes this time, mostly to address issues found by Lukas and
> > Marek.
> > 
> > The most significant one is to make device_link_add() cope with the case
> > when
> > the consumer device has not been registered yet when it is called.  The
> > supplier device still is required to be registered and the function will
> > return NULL if that is not the case.
> > 
> > Another significant change is in patch [4/5] that now makes the core apply
> > pm_runtime_get_sync()/pm_runtime_put() to supplier devices around the
> > probing of a consumer one (in analogy with the parent).
> 
> One more update after some conversations during LinuxCon Europe.
> 
> The main point was to make it possible for device_link_add() to figure out
> the initial state of the link instead of expecting the caller to provide it
> which might not be reliable enough in general.
> 
> In this version device_link_add() takes three arguments, the supplier and
> consumer pointers and flags and it sets the correct initial state of the
> link automatically (unless invoked with the "stateless" flag, of course).
> The cost is one additional field in struct device (I moved all of the
> links-related fields in struct device to a separate sub-structure while at
> it) to track the "driver presence status" of the device (to be used by
> device_link_add()).
> 
> In addition to that, the links list walks in the core.c and dd.c code are
> under the device links mutex now, so the iternal link spinlock is not needed
> any more and I have renamed symbols to distinguish between flags, link
> states and device "driver presence statuses".

The most significant change in this revision with respect to the previous one is
related to the fact that SRCU is not available on some architectures, so the
code falls back to using an RW semaphore for synchronization if SRCU is not
there.  Fortunately, the code changes needed for that turned out to be quite
straightforward and confined to the second patch.

Apart from this, the flags are defined using BIT(x) now (instead of open coding
the latter in the flag definitions).

Updated is mostly patch [2/5].  Patches [1,3,5/5] have not changed (except for
trivial rebasing) and patch [4/5] needed to be refreshed on top of the modified
[2/5].

FWIW, I've run the series through 0-day which has not reported any problems
with it.

Thanks,
Rafael

[Resend][PATCH v6 3/5] PM / sleep: Make async suspend/resume of devices use device links

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Make the device suspend/resume part of the core system
suspend/resume code use device links to ensure that supplier
and consumer devices will be suspended and resumed in the right
order in case of async suspend/resume.

The idea, roughly, is to use dpm_wait() to wait for all consumers
before a supplier device suspend and to wait for all suppliers
before a consumer device resume.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
---
 drivers/base/power/main.c |   85 ++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 79 insertions(+), 6 deletions(-)

Index: linux-pm/drivers/base/power/main.c
===================================================================
--- linux-pm.orig/drivers/base/power/main.c
+++ linux-pm/drivers/base/power/main.c
@@ -246,6 +246,62 @@ static void dpm_wait_for_children(struct
        device_for_each_child(dev, &async, dpm_wait_fn);
 }
 
+static void dpm_wait_for_suppliers(struct device *dev, bool async)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	/*
+	 * If the supplier goes away right after we've checked the link to it,
+	 * we'll wait for its completion to change the state, but that's fine,
+	 * because the only things that will block as a result are the SRCU
+	 * callbacks freeing the link objects for the links in the list we're
+	 * walking.
+	 */
+	list_for_each_entry_rcu(link, &dev->links.suppliers, c_node)
+		if (READ_ONCE(link->status) != DL_STATE_DORMANT)
+			dpm_wait(link->supplier, async);
+
+	device_links_read_unlock(idx);
+}
+
+static void dpm_wait_for_superior(struct device *dev, bool async)
+{
+	dpm_wait(dev->parent, async);
+	dpm_wait_for_suppliers(dev, async);
+}
+
+static void dpm_wait_for_consumers(struct device *dev, bool async)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	/*
+	 * The status of a device link can only be changed from "dormant" by a
+	 * probe, but that cannot happen during system suspend/resume.  In
+	 * theory it can change to "dormant" at that time, but then it is
+	 * reasonable to wait for the target device anyway (eg. if it goes
+	 * away, it's better to wait for it to go away completely and then
+	 * continue instead of trying to continue in parallel with its
+	 * unregistration).
+	 */
+	list_for_each_entry_rcu(link, &dev->links.consumers, s_node)
+		if (READ_ONCE(link->status) != DL_STATE_DORMANT)
+			dpm_wait(link->consumer, async);
+
+	device_links_read_unlock(idx);
+}
+
+static void dpm_wait_for_subordinate(struct device *dev, bool async)
+{
+	dpm_wait_for_children(dev, async);
+	dpm_wait_for_consumers(dev, async);
+}
+
 /**
  * pm_op - Return the PM operation appropriate for given PM event.
  * @ops: PM operations to choose from.
@@ -490,7 +546,7 @@ static int device_resume_noirq(struct de
 	if (!dev->power.is_noirq_suspended)
 		goto Out;
 
-	dpm_wait(dev->parent, async);
+	dpm_wait_for_superior(dev, async);
 
 	if (dev->pm_domain) {
 		info = "noirq power domain ";
@@ -620,7 +676,7 @@ static int device_resume_early(struct de
 	if (!dev->power.is_late_suspended)
 		goto Out;
 
-	dpm_wait(dev->parent, async);
+	dpm_wait_for_superior(dev, async);
 
 	if (dev->pm_domain) {
 		info = "early power domain ";
@@ -752,7 +808,7 @@ static int device_resume(struct device *
 		goto Complete;
 	}
 
-	dpm_wait(dev->parent, async);
+	dpm_wait_for_superior(dev, async);
 	dpm_watchdog_set(&wd, dev);
 	device_lock(dev);
 
@@ -1040,7 +1096,7 @@ static int __device_suspend_noirq(struct
 	if (dev->power.syscore || dev->power.direct_complete)
 		goto Complete;
 
-	dpm_wait_for_children(dev, async);
+	dpm_wait_for_subordinate(dev, async);
 
 	if (dev->pm_domain) {
 		info = "noirq power domain ";
@@ -1187,7 +1243,7 @@ static int __device_suspend_late(struct
 	if (dev->power.syscore || dev->power.direct_complete)
 		goto Complete;
 
-	dpm_wait_for_children(dev, async);
+	dpm_wait_for_subordinate(dev, async);
 
 	if (dev->pm_domain) {
 		info = "late power domain ";
@@ -1344,6 +1400,22 @@ static int legacy_suspend(struct device
 	return error;
 }
 
+static void dpm_clear_suppliers_direct_complete(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links.suppliers, c_node) {
+		spin_lock_irq(&link->supplier->power.lock);
+		link->supplier->power.direct_complete = false;
+		spin_unlock_irq(&link->supplier->power.lock);
+	}
+
+	device_links_read_unlock(idx);
+}
+
 /**
  * device_suspend - Execute "suspend" callbacks for given device.
  * @dev: Device to handle.
@@ -1360,7 +1432,7 @@ static int __device_suspend(struct devic
 	TRACE_DEVICE(dev);
 	TRACE_SUSPEND(0);
 
-	dpm_wait_for_children(dev, async);
+	dpm_wait_for_subordinate(dev, async);
 
 	if (async_error)
 		goto Complete;
@@ -1456,6 +1528,7 @@ static int __device_suspend(struct devic
 
 			spin_unlock_irq(&parent->power.lock);
 		}
+		dpm_clear_suppliers_direct_complete(dev);
 	}
 
 	device_unlock(dev);

[Resend][PATCH v6 1/5] driver core: Add a wrapper around __device_release_driver()

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Add an internal wrapper around __device_release_driver() that will
acquire device locks and do the necessary checks before calling it.

The next patch will make use of it.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
---
 drivers/base/dd.c |   30 ++++++++++++++++++------------
 1 file changed, 18 insertions(+), 12 deletions(-)

Index: linux-pm/drivers/base/dd.c
===================================================================
--- linux-pm.orig/drivers/base/dd.c
+++ linux-pm/drivers/base/dd.c
@@ -811,6 +811,22 @@ static void __device_release_driver(stru
 	}
 }
 
+static void device_release_driver_internal(struct device *dev,
+					   struct device_driver *drv,
+					   struct device *parent)
+{
+	if (parent)
+		device_lock(parent);
+
+	device_lock(dev);
+	if (!drv || drv == dev->driver)
+		__device_release_driver(dev);
+
+	device_unlock(dev);
+	if (parent)
+		device_unlock(parent);
+}
+
 /**
  * device_release_driver - manually detach device from driver.
  * @dev: device.
@@ -825,9 +841,7 @@ void device_release_driver(struct device
 	 * within their ->remove callback for the same device, they
 	 * will deadlock right here.
 	 */
-	device_lock(dev);
-	__device_release_driver(dev);
-	device_unlock(dev);
+	device_release_driver_internal(dev, NULL, NULL);
 }
 EXPORT_SYMBOL_GPL(device_release_driver);
 
@@ -852,15 +866,7 @@ void driver_detach(struct device_driver
 		dev = dev_prv->device;
 		get_device(dev);
 		spin_unlock(&drv->p->klist_devices.k_lock);
-
-		if (dev->parent)	/* Needed for USB */
-			device_lock(dev->parent);
-		device_lock(dev);
-		if (dev->driver == drv)
-			__device_release_driver(dev);
-		device_unlock(dev);
-		if (dev->parent)
-			device_unlock(dev->parent);
+		device_release_driver_internal(dev, drv, dev->parent);
 		put_device(dev);
 	}
 }

[PATCH v6 2/5] driver core: Functional dependencies tracking support

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Currently, there is a problem with taking functional dependencies
between devices into account.

What I mean by a "functional dependency" is when the driver of device
B needs device A to be functional and (generally) its driver to be
present in order to work properly.  This has certain consequences
for power management (suspend/resume and runtime PM ordering) and
shutdown ordering of these devices.  In general, it also implies that
the driver of A needs to be working for B to be probed successfully
and it cannot be unbound from the device before the B's driver.

Support for representing those functional dependencies between
devices is added here to allow the driver core to track them and act
on them in certain cases where applicable.

The argument for doing that in the driver core is that there are
quite a few distinct use cases involving device dependencies, they
are relatively hard to get right in a driver (if one wants to
address all of them properly) and it only gets worse if multiplied
by the number of drivers potentially needing to do it.  Morever, at
least one case (asynchronous system suspend/resume) cannot be handled
in a single driver at all, because it requires the driver of A to
wait for B to suspend (during system suspend) and the driver of B to
wait for A to resume (during system resume).

For this reason, represent dependencies between devices as "links",
with the help of struct device_link objects each containing pointers
to the "linked" devices, a list node for each of them, status
information, flags, and an RCU head for synchronization.

Also add two new list heads, representing the lists of links to the
devices that depend on the given one (consumers) and to the devices
depended on by it (suppliers), and a "driver presence status" field
(needed for figuring out initial states of device links) to struct
device.

The entire data structure consisting of all of the lists of link
objects for all devices is protected by a mutex (for link object
addition/removal and for list walks during device driver probing
and removal) and by SRCU (for list walking in other case that will
be introduced by subsequent change sets).  If CONFIG_SRCU is not
selected, however, an rwsem is used for protecting the entire data
structure.

In addition, each link object has an internal status field whose
value reflects whether or not drivers are bound to the devices
pointed to by the link or probing/removal of their drivers is in
progress etc.  That field is only modified under the device links
mutex, but it may be read outside of it in some cases (introduced by
subsequent change sets), so modifications of it are annotated with
WRITE_ONCE().

New links are added by calling device_link_add() which takes three
arguments: pointers to the devices in question and flags.  In
particular, if DL_FLAG_STATELESS is set in the flags, the link status
is not to be taken into account for this link and the driver core
will not manage it.  In turn, if DL_FLAG_AUTOREMOVE is set in the
flags, the driver core will remove the link automatically when the
consumer device driver unbinds from it.

One of the actions carried out by device_link_add() is to reorder
the lists used for device shutdown and system suspend/resume to
put the consumer device along with all of its children and all of
its consumers (and so on, recursively) to the ends of those lists
in order to ensure the right ordering between all of the supplier
and consumer devices.

For this reason, it is not possible to create a link between two
devices if the would-be supplier device already depends on the
would-be consumer device as either a direct descendant of it or a
consumer of one of its direct descendants or one of its consumers
and so on.

There are two types of link objects, persistent and non-persistent.
The persistent ones stay around until one of the target devices is
deleted, while the non-persistent ones are removed automatically when
the consumer driver unbinds from its device (ie. they are assumed to
be valid only as long as the consumer device has a driver bound to
it).  Persistent links are created by default and non-persistent
links are created when the DL_FLAG_AUTOREMOVE flag is passed
to device_link_add().

Both persistent and non-persistent device links can be deleted
with an explicit call to device_link_del().

Links created without the DL_FLAG_STATELESS flag set are managed
by the driver core using a simple state machine.  There are 5 states
each link can be in: DORMANT (unused), AVAILABLE (the supplier driver
is present and functional), CONSUMER_PROBE (the consumer driver is
probing), ACTIVE (both supplier and consumer drivers are present and
functional), and SUPPLIER_UNBIND (the supplier driver is unbinding).
The driver core updates the link state automatically depending on
what happens to the linked devices and for each link state specific
actions are taken in addition to that.

For example, if the supplier driver unbinds from its device, the
driver core will also unbind the drivers of all of its consumers
automatically under the assumption that they cannot function
properly without the supplier.  Analogously, the driver core will
only allow the consumer driver to bind to its device if the
supplier driver is present and functional (ie. the link is in
the AVAILABLE state).  If that's not the case, it will rely on
the existing deferred probing mechanism to wait for the supplier
driver to become available.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---

v5 -> v6:

- Fall back to using an rwsem for protecting the device links data structure
  if SRCU is not available (mutex+SRCU is used as before if CONFIG_SRCU is set,
  so the code should have not changed on systems with SRCU selected).

- Use BIT(x) in flag definitions.

v4 -> v5:

- Redefine device_link_add() to take three arguments, the supplier and consumer
  pointers and flags, and to figure out the initial link state automatically.

- Move the links-related fields in struct device to a separate sub-structure
  and add a "driver presence tracking" field to it (to help device_link_add()
  to do its job).

- Modify device_links_check_suppliers(), device_links_driver_bound(),
  device_links_no_driver(), device_links_driver_cleanup(), device_links_busy(),
  and device_links_unbind_consumers() to walk link lists under device_links_lock
  (to make the new "driver presence tracking" mechanism work reliably).

- Drop the (not necessary any more) spinlock from struct device_link.

- Rename symbols to better reflect their purpose (flags vs link states etc).

v3 -> v4:

- Add the in_dpm_list field to struct dev_pm_info and use it for checking
  if the device has been added to dpm_list already, which is needed for
  handling the "not registered consumer" case in device_link_add().

- Rework device_link_add() to handle consumer devices that have not been
  registered before calling it.

- Drop the parent check from device_link_add().

- Rename device_links_driver_gone() to device_links_driver_cleanup()

- Rearrange device_links_unbind_consumers() to avoid one spin_unlock()
  instance (which wasn't necessary).

- Introduce device_links_purge() to delete links from a device going away
  and make device_del() call it (instead of running links-related code
  directly).

- Move the invocation of device_links_ckeck_suppliers() to really_probe().

- Change the description of the DEVICE_LINK_STATELESS flag.

---
 drivers/base/base.h        |   13 +
 drivers/base/core.c        |  540 +++++++++++++++++++++++++++++++++++++++++++++
 drivers/base/dd.c          |   41 +++
 drivers/base/power/main.c  |    2 
 drivers/base/power/power.h |   10 
 include/linux/device.h     |   80 ++++++
 include/linux/pm.h         |    1 
 7 files changed, 682 insertions(+), 5 deletions(-)

Index: linux-pm/include/linux/pm.h
===================================================================
--- linux-pm.orig/include/linux/pm.h
+++ linux-pm/include/linux/pm.h
@@ -559,6 +559,7 @@ struct dev_pm_info {
 	pm_message_t		power_state;
 	unsigned int		can_wakeup:1;
 	unsigned int		async_suspend:1;
+	bool			in_dpm_list:1;	/* Owned by the PM core */
 	bool			is_prepared:1;	/* Owned by the PM core */
 	bool			is_suspended:1;	/* Ditto */
 	bool			is_noirq_suspended:1;
Index: linux-pm/drivers/base/power/power.h
===================================================================
--- linux-pm.orig/drivers/base/power/power.h
+++ linux-pm/drivers/base/power/power.h
@@ -127,6 +127,11 @@ extern void device_pm_move_after(struct
 extern void device_pm_move_last(struct device *);
 extern void device_pm_check_callbacks(struct device *dev);
 
+static inline bool device_pm_initialized(struct device *dev)
+{
+	return dev->power.in_dpm_list;
+}
+
 #else /* !CONFIG_PM_SLEEP */
 
 static inline void device_pm_sleep_init(struct device *dev) {}
@@ -146,6 +151,11 @@ static inline void device_pm_move_last(s
 
 static inline void device_pm_check_callbacks(struct device *dev) {}
 
+static inline bool device_pm_initialized(struct device *dev)
+{
+	return device_is_registered(dev);
+}
+
 #endif /* !CONFIG_PM_SLEEP */
 
 static inline void device_pm_init(struct device *dev)
Index: linux-pm/drivers/base/power/main.c
===================================================================
--- linux-pm.orig/drivers/base/power/main.c
+++ linux-pm/drivers/base/power/main.c
@@ -131,6 +131,7 @@ void device_pm_add(struct device *dev)
 		dev_warn(dev, "parent %s should not be sleeping\n",
 			dev_name(dev->parent));
 	list_add_tail(&dev->power.entry, &dpm_list);
+	dev->power.in_dpm_list = true;
 	mutex_unlock(&dpm_list_mtx);
 }
 
@@ -145,6 +146,7 @@ void device_pm_remove(struct device *dev
 	complete_all(&dev->power.completion);
 	mutex_lock(&dpm_list_mtx);
 	list_del_init(&dev->power.entry);
+	dev->power.in_dpm_list = false;
 	mutex_unlock(&dpm_list_mtx);
 	device_wakeup_disable(dev);
 	pm_runtime_remove(dev);
Index: linux-pm/include/linux/device.h
===================================================================
--- linux-pm.orig/include/linux/device.h
+++ linux-pm/include/linux/device.h
@@ -708,6 +708,81 @@ struct device_dma_parameters {
 };
 
 /**
+ * enum device_link_state - Device link states.
+ * @DL_STATE_NONE: The presence of the drivers is not being tracked.
+ * @DL_STATE_DORMANT: None of the supplier/consumer drivers is present.
+ * @DL_STATE_AVAILABLE: The supplier driver is present, but the consumer is not.
+ * @DL_STATE_CONSUMER_PROBE: The consumer is probing (supplier driver present).
+ * @DL_STATE_ACTIVE: Both the supplier and consumer drivers are present.
+ * @DL_STATE_SUPPLIER_UNBIND: The supplier driver is unbinding.
+ */
+enum device_link_state {
+	DL_STATE_NONE = -1,
+	DL_STATE_DORMANT = 0,
+	DL_STATE_AVAILABLE,
+	DL_STATE_CONSUMER_PROBE,
+	DL_STATE_ACTIVE,
+	DL_STATE_SUPPLIER_UNBIND,
+};
+
+/*
+ * Device link flags.
+ *
+ * STATELESS: The core won't track the presence of supplier/consumer drivers.
+ * AUTOREMOVE: Remove this link automatically on consumer driver unbind.
+ */
+#define DL_FLAG_STATELESS	BIT(0)
+#define DL_FLAG_AUTOREMOVE	BIT(1)
+
+/**
+ * struct device_link - Device link representation.
+ * @supplier: The device on the supplier end of the link.
+ * @s_node: Hook to the supplier device's list of links to consumers.
+ * @consumer: The device on the consumer end of the link.
+ * @c_node: Hook to the consumer device's list of links to suppliers.
+ * @status: The state of the link (with respect to the presence of drivers).
+ * @flags: Link flags.
+ * @rcu_head: An RCU head to use for deferred execution of SRCU callbacks.
+ */
+struct device_link {
+	struct device *supplier;
+	struct list_head s_node;
+	struct device *consumer;
+	struct list_head c_node;
+	enum device_link_state status;
+	u32 flags;
+#ifdef CONFIG_SRCU
+	struct rcu_head rcu_head;
+#endif
+};
+
+/**
+ * enum dl_dev_state - Device driver presence tracking information.
+ * @DL_DEV_NO_DRIVER: There is no driver attached to the device.
+ * @DL_DEV_PROBING: A driver is probing.
+ * @DL_DEV_DRIVER_BOUND: The driver has been bound to the device.
+ * @DL_DEV_UNBINDING: The driver is unbinding from the device.
+ */
+enum dl_dev_state {
+	DL_DEV_NO_DRIVER = 0,
+	DL_DEV_PROBING,
+	DL_DEV_DRIVER_BOUND,
+	DL_DEV_UNBINDING,
+};
+
+/**
+ * struct dev_links_info - Device data related to device links.
+ * @suppliers: List of links to supplier devices.
+ * @consumers: List of links to consumer devices.
+ * @status: Driver status information.
+ */
+struct dev_links_info {
+	struct list_head suppliers;
+	struct list_head consumers;
+	enum dl_dev_state status;
+};
+
+/**
  * struct device - The basic device structure
  * @parent:	The device's "parent" device, the device to which it is attached.
  * 		In most cases, a parent device is some sort of bus or host
@@ -799,6 +874,7 @@ struct device {
 					   core doesn't touch it */
 	void		*driver_data;	/* Driver data, set and get with
 					   dev_set/get_drvdata */
+	struct dev_links_info	links;
 	struct dev_pm_info	power;
 	struct dev_pm_domain	*pm_domain;
 
@@ -1116,6 +1192,10 @@ extern void device_shutdown(void);
 /* debugging and troubleshooting/diagnostic helpers. */
 extern const char *dev_driver_string(const struct device *dev);
 
+/* Device links interface. */
+struct device_link *device_link_add(struct device *consumer,
+				    struct device *supplier, u32 flags);
+void device_link_del(struct device_link *link);
 
 #ifdef CONFIG_PRINTK
 
Index: linux-pm/drivers/base/base.h
===================================================================
--- linux-pm.orig/drivers/base/base.h
+++ linux-pm/drivers/base/base.h
@@ -107,6 +107,9 @@ extern void bus_remove_device(struct dev
 
 extern int bus_add_driver(struct device_driver *drv);
 extern void bus_remove_driver(struct device_driver *drv);
+extern void device_release_driver_internal(struct device *dev,
+					   struct device_driver *drv,
+					   struct device *parent);
 
 extern void driver_detach(struct device_driver *drv);
 extern int driver_probe_device(struct device_driver *drv, struct device *dev);
@@ -152,3 +155,13 @@ extern int devtmpfs_init(void);
 #else
 static inline int devtmpfs_init(void) { return 0; }
 #endif
+
+/* Device links support */
+extern int device_links_read_lock(void);
+extern void device_links_read_unlock(int idx);
+extern int device_links_check_suppliers(struct device *dev);
+extern void device_links_driver_bound(struct device *dev);
+extern void device_links_driver_cleanup(struct device *dev);
+extern void device_links_no_driver(struct device *dev);
+extern bool device_links_busy(struct device *dev);
+extern void device_links_unbind_consumers(struct device *dev);
Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -44,6 +44,541 @@ static int __init sysfs_deprecated_setup
 early_param("sysfs.deprecated", sysfs_deprecated_setup);
 #endif
 
+/* Device links support. */
+
+#ifdef CONFIG_SRCU
+static DEFINE_MUTEX(device_links_lock);
+DEFINE_STATIC_SRCU(device_links_srcu);
+
+static inline void device_links_write_lock(void)
+{
+	mutex_lock(&device_links_lock);
+}
+
+static inline void device_links_write_unlock(void)
+{
+	mutex_unlock(&device_links_lock);
+}
+
+int device_links_read_lock(void)
+{
+	return srcu_read_lock(&device_links_srcu);
+}
+
+void device_links_read_unlock(int idx)
+{
+	srcu_read_unlock(&device_links_srcu, idx);
+}
+#else /* !CONFIG_SRCU */
+static DECLARE_RWSEM(device_links_lock);
+
+static inline void device_links_write_lock(void)
+{
+	down_write(&device_links_lock);
+}
+
+static inline void device_links_write_unlock(void)
+{
+	up_write(&device_links_lock);
+}
+
+int device_links_read_lock(void)
+{
+	down_read(&device_links_lock);
+	return 0;
+}
+
+void device_links_read_unlock(int not_used)
+{
+	up_read(&device_links_lock);
+}
+#endif /* !CONFIG_SRCU */
+
+/**
+ * device_is_dependent - Check if one device depends on another one
+ * @dev: Device to check dependencies for.
+ * @target: Device to check against.
+ *
+ * Check if @target depends on @dev or any device dependent on it (its child or
+ * its consumer etc).  Return 1 if that is the case or 0 otherwise.
+ */
+static int device_is_dependent(struct device *dev, void *target)
+{
+	struct device_link *link;
+	int ret;
+
+	if (WARN_ON(dev == target))
+		return 1;
+
+	ret = device_for_each_child(dev, target, device_is_dependent);
+	if (ret)
+		return ret;
+
+	list_for_each_entry(link, &dev->links.consumers, s_node) {
+		if (WARN_ON(link->consumer == target))
+			return 1;
+
+		ret = device_is_dependent(link->consumer, target);
+		if (ret)
+			break;
+	}
+	return ret;
+}
+
+static int device_reorder_to_tail(struct device *dev, void *not_used)
+{
+	struct device_link *link;
+
+	/*
+	 * Devices that have not been registered yet will be put to the ends
+	 * of the lists during the registration, so skip them here.
+	 */
+	if (device_is_registered(dev))
+		devices_kset_move_last(dev);
+
+	if (device_pm_initialized(dev))
+		device_pm_move_last(dev);
+
+	device_for_each_child(dev, NULL, device_reorder_to_tail);
+	list_for_each_entry(link, &dev->links.consumers, s_node)
+		device_reorder_to_tail(link->consumer, NULL);
+
+	return 0;
+}
+
+/**
+ * device_link_add - Create a link between two devices.
+ * @consumer: Consumer end of the link.
+ * @supplier: Supplier end of the link.
+ * @flags: Link flags.
+ *
+ * If the DL_FLAG_AUTOREMOVE is set, the link will be removed automatically
+ * when the consumer device driver unbinds from it.  The combination of both
+ * DL_FLAG_AUTOREMOVE and DL_FLAG_STATELESS set is invalid and will cause NULL
+ * to be returned.
+ *
+ * A side effect of the link creation is re-ordering of dpm_list and the
+ * devices_kset list by moving the consumer device and all devices depending
+ * on it to the ends of these lists (that does not happen to devices that have
+ * not been registered when this function is called).
+ *
+ * The supplier device is required to be registered when this function is called
+ * and NULL will be returned if that is not the case.  The consumer device need
+ * not be registerd, however.
+ */
+struct device_link *device_link_add(struct device *consumer,
+				    struct device *supplier, u32 flags)
+{
+	struct device_link *link;
+
+	if (!consumer || !supplier ||
+	    ((flags & DL_FLAG_STATELESS) && (flags & DL_FLAG_AUTOREMOVE)))
+		return NULL;
+
+	device_links_write_lock();
+	device_pm_lock();
+
+	/*
+	 * If the supplier has not been fully registered yet or there is a
+	 * reverse dependency between the consumer and the supplier already in
+	 * the graph, return NULL.
+	 */
+	if (!device_pm_initialized(supplier)
+	    || device_is_dependent(consumer, supplier)) {
+		link = NULL;
+		goto out;
+	}
+
+	list_for_each_entry(link, &supplier->links.consumers, s_node)
+		if (link->consumer == consumer)
+			goto out;
+
+	link = kmalloc(sizeof(*link), GFP_KERNEL);
+	if (!link)
+		goto out;
+
+	get_device(supplier);
+	link->supplier = supplier;
+	INIT_LIST_HEAD(&link->s_node);
+	get_device(consumer);
+	link->consumer = consumer;
+	INIT_LIST_HEAD(&link->c_node);
+	link->flags = flags;
+
+	/* Deterine the initial link state. */
+	if (flags & DL_FLAG_STATELESS) {
+		link->status = DL_STATE_NONE;
+	} else {
+		switch (supplier->links.status) {
+		case DL_DEV_DRIVER_BOUND:
+			switch (consumer->links.status) {
+			case DL_DEV_PROBING:
+				link->status = DL_STATE_CONSUMER_PROBE;
+				break;
+			case DL_DEV_DRIVER_BOUND:
+				link->status = DL_STATE_ACTIVE;
+				break;
+			default:
+				link->status = DL_STATE_AVAILABLE;
+				break;
+			}
+			break;
+		case DL_DEV_UNBINDING:
+			link->status = DL_STATE_SUPPLIER_UNBIND;
+			break;
+		default:
+			link->status = DL_STATE_DORMANT;
+			break;
+		}
+	}
+
+	/*
+	 * Move the consumer and all of the devices depending on it to the end
+	 * of dpm_list and the devices_kset list.
+	 *
+	 * It is necessary to hold dpm_list locked throughout all that or else
+	 * we may end up suspending with a wrong ordering of it.
+	 */
+	device_reorder_to_tail(consumer, NULL);
+
+	list_add_tail_rcu(&link->s_node, &supplier->links.consumers);
+	list_add_tail_rcu(&link->c_node, &consumer->links.suppliers);
+
+	dev_info(consumer, "Linked as a consumer to %s\n", dev_name(supplier));
+
+ out:
+	device_pm_unlock();
+	device_links_write_unlock();
+	return link;
+}
+EXPORT_SYMBOL_GPL(device_link_add);
+
+static void device_link_free(struct device_link *link)
+{
+	put_device(link->consumer);
+	put_device(link->supplier);
+	kfree(link);
+}
+
+#ifdef CONFIG_SRCU
+static void __device_link_free_srcu(struct rcu_head *rhead)
+{
+	device_link_free(container_of(rhead, struct device_link, rcu_head));
+}
+
+static void __device_link_del(struct device_link *link)
+{
+	dev_info(link->consumer, "Dropping the link to %s\n",
+		 dev_name(link->supplier));
+
+	list_del_rcu(&link->s_node);
+	list_del_rcu(&link->c_node);
+	call_srcu(&device_links_srcu, &link->rcu_head, __device_link_free_srcu);
+}
+#else /* !CONFIG_SRCU */
+static void __device_link_del(struct device_link *link)
+{
+	dev_info(link->consumer, "Dropping the link to %s\n",
+		 dev_name(link->supplier));
+
+	list_del(&link->s_node);
+	list_del(&link->c_node);
+	device_link_free(link);
+}
+#endif /* !CONFIG_SRCU */
+
+/**
+ * device_link_del - Delete a link between two devices.
+ * @link: Device link to delete.
+ *
+ * The caller must ensure proper synchronization of this function with runtime
+ * PM.
+ */
+void device_link_del(struct device_link *link)
+{
+	device_links_write_lock();
+	device_pm_lock();
+	__device_link_del(link);
+	device_pm_unlock();
+	device_links_write_unlock();
+}
+EXPORT_SYMBOL_GPL(device_link_del);
+
+static void device_links_missing_supplier(struct device *dev)
+{
+	struct device_link *link;
+
+	list_for_each_entry(link, &dev->links.suppliers, c_node)
+		if (link->status == DL_STATE_CONSUMER_PROBE)
+			WRITE_ONCE(link->status, DL_STATE_AVAILABLE);
+}
+
+/**
+ * device_links_check_suppliers - Check presence of supplier drivers.
+ * @dev: Consumer device.
+ *
+ * Check links from this device to any suppliers.  Walk the list of the device's
+ * links to suppliers and see if all of them are available.  If not, simply
+ * return -EPROBE_DEFER.
+ *
+ * We need to guarantee that the supplier will not go away after the check has
+ * been positive here.  It only can go away in __device_release_driver() and
+ * that function  checks the device's links to consumers.  This means we need to
+ * mark the link as "consumer probe in progress" to make the supplier removal
+ * wait for us to complete (or bad things may happen).
+ *
+ * Links with the DL_FLAG_STATELESS flag set are ignored.
+ */
+int device_links_check_suppliers(struct device *dev)
+{
+	struct device_link *link;
+	int ret = 0;
+
+	device_links_write_lock();
+
+	list_for_each_entry(link, &dev->links.suppliers, c_node) {
+		if (link->flags & DL_FLAG_STATELESS)
+			continue;
+
+		if (link->status != DL_STATE_AVAILABLE) {
+			device_links_missing_supplier(dev);
+			ret = -EPROBE_DEFER;
+			break;
+		}
+		WRITE_ONCE(link->status, DL_STATE_CONSUMER_PROBE);
+	}
+	dev->links.status = DL_DEV_PROBING;
+
+	device_links_write_unlock();
+	return ret;
+}
+
+/**
+ * device_links_driver_bound - Update device links after probing its driver.
+ * @dev: Device to update the links for.
+ *
+ * The probe has been successful, so update links from this device to any
+ * consumers by changing their status to "available".
+ *
+ * Also change the status of @dev's links to suppliers to "active".
+ *
+ * Links with the DL_FLAG_STATELESS flag set are ignored.
+ */
+void device_links_driver_bound(struct device *dev)
+{
+	struct device_link *link;
+
+	device_links_write_lock();
+
+	list_for_each_entry(link, &dev->links.consumers, s_node) {
+		if (link->flags & DL_FLAG_STATELESS)
+			continue;
+
+		WARN_ON(link->status != DL_STATE_DORMANT);
+		WRITE_ONCE(link->status, DL_STATE_AVAILABLE);
+	}
+
+	list_for_each_entry(link, &dev->links.suppliers, c_node) {
+		if (link->flags & DL_FLAG_STATELESS)
+			continue;
+
+		WARN_ON(link->status != DL_STATE_CONSUMER_PROBE);
+		WRITE_ONCE(link->status, DL_STATE_ACTIVE);
+	}
+
+	dev->links.status = DL_DEV_DRIVER_BOUND;
+
+	device_links_write_unlock();
+}
+
+/**
+ * __device_links_no_driver - Update links of a device without a driver.
+ * @dev: Device without a drvier.
+ *
+ * Delete all non-persistent links from this device to any suppliers.
+ *
+ * Persistent links stay around, but their status is changed to "available",
+ * unless they already are in the "supplier unbind in progress" state in which
+ * case they need not be updated.
+ *
+ * Links with the DL_FLAG_STATELESS flag set are ignored.
+ */
+static void __device_links_no_driver(struct device *dev)
+{
+	struct device_link *link, *ln;
+
+	list_for_each_entry_safe_reverse(link, ln, &dev->links.suppliers, c_node) {
+		if (link->flags & DL_FLAG_STATELESS)
+			continue;
+
+		if (link->flags & DL_FLAG_AUTOREMOVE)
+			__device_link_del(link);
+		else if (link->status != DL_STATE_SUPPLIER_UNBIND)
+			WRITE_ONCE(link->status, DL_STATE_AVAILABLE);
+	}
+
+	dev->links.status = DL_DEV_NO_DRIVER;
+}
+
+void device_links_no_driver(struct device *dev)
+{
+	device_links_write_lock();
+	__device_links_no_driver(dev);
+	device_links_write_unlock();
+}
+
+/**
+ * device_links_driver_cleanup - Update links after driver removal.
+ * @dev: Device whose driver has just gone away.
+ *
+ * Update links to consumers for @dev by changing their status to "dormant" and
+ * invoke %__device_links_no_driver() to update links to suppliers for it as
+ * appropriate.
+ *
+ * Links with the DL_FLAG_STATELESS flag set are ignored.
+ */
+void device_links_driver_cleanup(struct device *dev)
+{
+	struct device_link *link;
+
+	device_links_write_lock();
+
+	list_for_each_entry(link, &dev->links.consumers, s_node) {
+		if (link->flags & DL_FLAG_STATELESS)
+			continue;
+
+		WARN_ON(link->flags & DL_FLAG_AUTOREMOVE);
+		WARN_ON(link->status != DL_STATE_SUPPLIER_UNBIND);
+		WRITE_ONCE(link->status, DL_STATE_DORMANT);
+	}
+
+	__device_links_no_driver(dev);
+
+	device_links_write_unlock();
+}
+
+/**
+ * device_links_busy - Check if there are any busy links to consumers.
+ * @dev: Device to check.
+ *
+ * Check each consumer of the device and return 'true' if its link's status
+ * is one of "consumer probe" or "active" (meaning that the given consumer is
+ * probing right now or its driver is present).  Otherwise, change the link
+ * state to "supplier unbind" to prevent the consumer from being probed
+ * successfully going forward.
+ *
+ * Return 'false' if there are no probing or active consumers.
+ *
+ * Links with the DL_FLAG_STATELESS flag set are ignored.
+ */
+bool device_links_busy(struct device *dev)
+{
+	struct device_link *link;
+	bool ret = false;
+
+	device_links_write_lock();
+
+	list_for_each_entry(link, &dev->links.consumers, s_node) {
+		if (link->flags & DL_FLAG_STATELESS)
+			continue;
+
+		if (link->status == DL_STATE_CONSUMER_PROBE
+		    || link->status == DL_STATE_ACTIVE) {
+			ret = true;
+			break;
+		}
+		WRITE_ONCE(link->status, DL_STATE_SUPPLIER_UNBIND);
+	}
+
+	dev->links.status = DL_DEV_UNBINDING;
+
+	device_links_write_unlock();
+	return ret;
+}
+
+/**
+ * device_links_unbind_consumers - Force unbind consumers of the given device.
+ * @dev: Device to unbind the consumers of.
+ *
+ * Walk the list of links to consumers for @dev and if any of them is in the
+ * "consumer probe" state, wait for all device probes in progress to complete
+ * and start over.
+ *
+ * If that's not the case, change the status of the link to "supplier unbind"
+ * and check if the link was in the "active" state.  If so, force the consumer
+ * driver to unbind and start over (the consumer will not re-probe as we have
+ * changed the state of the link already).
+ *
+ * Links with the DL_FLAG_STATELESS flag set are ignored.
+ */
+void device_links_unbind_consumers(struct device *dev)
+{
+	struct device_link *link;
+
+ start:
+	device_links_write_lock();
+
+	list_for_each_entry(link, &dev->links.consumers, s_node) {
+		enum device_link_state status;
+
+		if (link->flags & DL_FLAG_STATELESS)
+			continue;
+
+		status = link->status;
+		if (status == DL_STATE_CONSUMER_PROBE) {
+			device_links_write_unlock();
+
+			wait_for_device_probe();
+			goto start;
+		}
+		WRITE_ONCE(link->status, DL_STATE_SUPPLIER_UNBIND);
+		if (status == DL_STATE_ACTIVE) {
+			struct device *consumer = link->consumer;
+
+			get_device(consumer);
+
+			device_links_write_unlock();
+
+			device_release_driver_internal(consumer, NULL,
+						       consumer->parent);
+			put_device(consumer);
+			goto start;
+		}
+	}
+
+	device_links_write_unlock();
+}
+
+/**
+ * device_links_purge - Delete existing links to other devices.
+ * @dev: Target device.
+ */
+static void device_links_purge(struct device *dev)
+{
+	struct device_link *link, *ln;
+
+	/*
+	 * Delete all of the remaining links from this device to any other
+	 * devices (either consumers or suppliers).
+	 */
+	device_links_write_lock();
+
+	list_for_each_entry_safe_reverse(link, ln, &dev->links.suppliers, c_node) {
+		WARN_ON(link->status == DL_STATE_ACTIVE);
+		__device_link_del(link);
+	}
+
+	list_for_each_entry_safe_reverse(link, ln, &dev->links.consumers, s_node) {
+		WARN_ON(link->status != DL_STATE_DORMANT &&
+			link->status != DL_STATE_NONE);
+		__device_link_del(link);
+	}
+
+	device_links_write_unlock();
+}
+
+/* Device links support end. */
+
 int (*platform_notify)(struct device *dev) = NULL;
 int (*platform_notify_remove)(struct device *dev) = NULL;
 static struct kobject *dev_kobj;
@@ -711,6 +1246,9 @@ void device_initialize(struct device *de
 #ifdef CONFIG_GENERIC_MSI_IRQ
 	INIT_LIST_HEAD(&dev->msi_list);
 #endif
+	INIT_LIST_HEAD(&dev->links.consumers);
+	INIT_LIST_HEAD(&dev->links.suppliers);
+	dev->links.status = DL_DEV_NO_DRIVER;
 }
 EXPORT_SYMBOL_GPL(device_initialize);
 
@@ -1258,6 +1796,8 @@ void device_del(struct device *dev)
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DEL_DEVICE, dev);
+
+	device_links_purge(dev);
 	dpm_sysfs_remove(dev);
 	if (parent)
 		klist_del(&dev->p->knode_parent);
Index: linux-pm/drivers/base/dd.c
===================================================================
--- linux-pm.orig/drivers/base/dd.c
+++ linux-pm/drivers/base/dd.c
@@ -244,6 +244,7 @@ static void driver_bound(struct device *
 		 __func__, dev_name(dev));
 
 	klist_add_tail(&dev->p->knode_driver, &dev->driver->p->klist_devices);
+	device_links_driver_bound(dev);
 
 	device_pm_check_callbacks(dev);
 
@@ -337,6 +338,10 @@ static int really_probe(struct device *d
 		return ret;
 	}
 
+	ret = device_links_check_suppliers(dev);
+	if (ret)
+		return ret;
+
 	atomic_inc(&probe_count);
 	pr_debug("bus: '%s': %s: probing driver %s with device %s\n",
 		 drv->bus->name, __func__, drv->name, dev_name(dev));
@@ -415,6 +420,7 @@ probe_failed:
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);
 pinctrl_bind_failed:
+	device_links_no_driver(dev);
 	devres_release_all(dev);
 	driver_sysfs_remove(dev);
 	dev->driver = NULL;
@@ -771,7 +777,7 @@ EXPORT_SYMBOL_GPL(driver_attach);
  * __device_release_driver() must be called with @dev lock held.
  * When called for a USB interface, @dev->parent lock must be held as well.
  */
-static void __device_release_driver(struct device *dev)
+static void __device_release_driver(struct device *dev, struct device *parent)
 {
 	struct device_driver *drv;
 
@@ -780,6 +786,25 @@ static void __device_release_driver(stru
 		if (driver_allows_async_probing(drv))
 			async_synchronize_full();
 
+		while (device_links_busy(dev)) {
+			device_unlock(dev);
+			if (parent)
+				device_unlock(parent);
+
+			device_links_unbind_consumers(dev);
+			if (parent)
+				device_lock(parent);
+
+			device_lock(dev);
+			/*
+			 * A concurrent invocation of the same function might
+			 * have released the driver successfully while this one
+			 * was waiting, so check for that.
+			 */
+			if (dev->driver != drv)
+				return;
+		}
+
 		pm_runtime_get_sync(dev);
 
 		driver_sysfs_remove(dev);
@@ -795,6 +820,8 @@ static void __device_release_driver(stru
 			dev->bus->remove(dev);
 		else if (drv->remove)
 			drv->remove(dev);
+
+		device_links_driver_cleanup(dev);
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);
@@ -811,16 +838,16 @@ static void __device_release_driver(stru
 	}
 }
 
-static void device_release_driver_internal(struct device *dev,
-					   struct device_driver *drv,
-					   struct device *parent)
+void device_release_driver_internal(struct device *dev,
+				    struct device_driver *drv,
+				    struct device *parent)
 {
 	if (parent)
 		device_lock(parent);
 
 	device_lock(dev);
 	if (!drv || drv == dev->driver)
-		__device_release_driver(dev);
+		__device_release_driver(dev, parent);
 
 	device_unlock(dev);
 	if (parent)
@@ -833,6 +860,10 @@ static void device_release_driver_intern
  *
  * Manually detach device from driver.
  * When called for a USB interface, @dev->parent lock must be held.
+ *
+ * If this function is to be called with @dev->parent lock held, ensure that
+ * the device's consumers are unbound in advance or that their locks can be
+ * acquired under the @dev->parent lock.
  */
 void device_release_driver(struct device *dev)
 {

[PATCH v6 4/5] PM / runtime: Use device links

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Modify the runtime PM framework to use device links to ensure that
supplier devices will not be suspended if any of their consumer
devices are active.

The idea is to reference count suppliers on the consumer's resume
and drop references to them on its suspend.  The information on
whether or not the supplier has been reference counted by the
consumer's (runtime) resume is stored in a new field (rpm_active)
in the link object for each link.

It may be necessary to clean up those references when the
supplier is unbinding and that's why the links whose status is
DEVICE_LINK_SUPPLIER_UNBIND are skipped by the runtime suspend
and resume code.

The above means that if the consumer device is probed in the
runtime-active state, the supplier has to be resumed and reference
counted by device_link_add() so the code works as expected on its
(runtime) suspend.  There is a new flag, DEVICE_LINK_RPM_ACTIVE,
to tell device_link_add() about that (in which case the caller
is responsible for making sure that the consumer really will
be runtime-active when runtime PM is enabled for it).

The other new link flag, DEVICE_LINK_PM_RUNTIME, tells the core
whether or not the link should be used for runtime PM at all.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---

v5 -> v6:

- Rebase.

- Use BIT(x) in flag definitions.

v4 -> v5:

- Rebase, change the symbol names in accordance with patch [2/5] etc.

- Use READ_ONCE() to annotate "unprotected" accesses to link status data.

v3 -> v4:

- Add pm_runtime_get/put_suppliers() and make the core call them
  around the probing of a consumer device (in analogy with what it
  does to the device's parent).

---
 drivers/base/core.c          |   27 +++++++
 drivers/base/dd.c            |    3 
 drivers/base/power/runtime.c |  157 +++++++++++++++++++++++++++++++++++++++++--
 include/linux/device.h       |    6 +
 include/linux/pm_runtime.h   |    6 +
 5 files changed, 193 insertions(+), 6 deletions(-)

Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -152,6 +152,14 @@ static int device_reorder_to_tail(struct
  * @supplier: Supplier end of the link.
  * @flags: Link flags.
  *
+ * The caller is responsible for the proper synchronization of the link creation
+ * with runtime PM.  First, setting the DL_FLAG_PM_RUNTIME flag will cause the
+ * runtime PM framework to take the link into account.  Second, if the
+ * DL_FLAG_RPM_ACTIVE flag is set in addition to it, the supplier devices will
+ * be forced into the active metastate and reference-counted upon the creation
+ * of the link.  If DL_FLAG_PM_RUNTIME is not set, DL_FLAG_RPM_ACTIVE will be
+ * ignored.
+ *
  * If the DL_FLAG_AUTOREMOVE is set, the link will be removed automatically
  * when the consumer device driver unbinds from it.  The combination of both
  * DL_FLAG_AUTOREMOVE and DL_FLAG_STATELESS set is invalid and will cause NULL
@@ -193,10 +201,19 @@ struct device_link *device_link_add(stru
 		if (link->consumer == consumer)
 			goto out;
 
-	link = kmalloc(sizeof(*link), GFP_KERNEL);
+	link = kzalloc(sizeof(*link), GFP_KERNEL);
 	if (!link)
 		goto out;
 
+	if ((flags & DL_FLAG_PM_RUNTIME) && (flags & DL_FLAG_RPM_ACTIVE)) {
+		if (pm_runtime_get_sync(supplier) < 0) {
+			pm_runtime_put_noidle(supplier);
+			kfree(link);
+			link = NULL;
+			goto out;
+		}
+		link->rpm_active = true;
+	}
 	get_device(supplier);
 	link->supplier = supplier;
 	INIT_LIST_HEAD(&link->s_node);
@@ -213,6 +230,14 @@ struct device_link *device_link_add(stru
 		case DL_DEV_DRIVER_BOUND:
 			switch (consumer->links.status) {
 			case DL_DEV_PROBING:
+				/*
+				 * Balance the decrementation of the supplier's
+				 * runtime PM usage counter after consumer probe
+				 * in driver_probe_device().
+				 */
+				if (flags & DL_FLAG_PM_RUNTIME)
+					pm_runtime_get_sync(supplier);
+
 				link->status = DL_STATE_CONSUMER_PROBE;
 				break;
 			case DL_DEV_DRIVER_BOUND:
Index: linux-pm/drivers/base/dd.c
===================================================================
--- linux-pm.orig/drivers/base/dd.c
+++ linux-pm/drivers/base/dd.c
@@ -513,6 +513,7 @@ int driver_probe_device(struct device_dr
 	pr_debug("bus: '%s': %s: matched device %s with driver %s\n",
 		 drv->bus->name, __func__, dev_name(dev), drv->name);
 
+	pm_runtime_get_suppliers(dev);
 	if (dev->parent)
 		pm_runtime_get_sync(dev->parent);
 
@@ -523,6 +524,7 @@ int driver_probe_device(struct device_dr
 	if (dev->parent)
 		pm_runtime_put(dev->parent);
 
+	pm_runtime_put_suppliers(dev);
 	return ret;
 }
 
@@ -806,6 +808,7 @@ static void __device_release_driver(stru
 		}
 
 		pm_runtime_get_sync(dev);
+		pm_runtime_clean_up_links(dev);
 
 		driver_sysfs_remove(dev);
 
Index: linux-pm/drivers/base/power/runtime.c
===================================================================
--- linux-pm.orig/drivers/base/power/runtime.c
+++ linux-pm/drivers/base/power/runtime.c
@@ -12,6 +12,8 @@
 #include <linux/pm_runtime.h>
 #include <linux/pm_wakeirq.h>
 #include <trace/events/rpm.h>
+
+#include "../base.h"
 #include "power.h"
 
 typedef int (*pm_callback_t)(struct device *);
@@ -258,6 +260,42 @@ static int rpm_check_suspend_allowed(str
 	return retval;
 }
 
+static int rpm_get_suppliers(struct device *dev)
+{
+	struct device_link *link;
+
+	list_for_each_entry_rcu(link, &dev->links.suppliers, c_node) {
+		int retval;
+
+		if (!(link->flags & DL_FLAG_PM_RUNTIME))
+			continue;
+
+		if (READ_ONCE(link->status) == DL_STATE_SUPPLIER_UNBIND ||
+		    link->rpm_active)
+			continue;
+
+		retval = pm_runtime_get_sync(link->supplier);
+		if (retval < 0) {
+			pm_runtime_put_noidle(link->supplier);
+			return retval;
+		}
+		link->rpm_active = true;
+	}
+	return 0;
+}
+
+static void rpm_put_suppliers(struct device *dev)
+{
+	struct device_link *link;
+
+	list_for_each_entry_rcu(link, &dev->links.suppliers, c_node)
+		if (link->rpm_active &&
+		    READ_ONCE(link->status) != DL_STATE_SUPPLIER_UNBIND) {
+			pm_runtime_put(link->supplier);
+			link->rpm_active = false;
+		}
+}
+
 /**
  * __rpm_callback - Run a given runtime PM callback for a given device.
  * @cb: Runtime PM callback to run.
@@ -266,19 +304,55 @@ static int rpm_check_suspend_allowed(str
 static int __rpm_callback(int (*cb)(struct device *), struct device *dev)
 	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
-	int retval;
+	int retval, idx;
 
-	if (dev->power.irq_safe)
+	if (dev->power.irq_safe) {
 		spin_unlock(&dev->power.lock);
-	else
+	} else {
 		spin_unlock_irq(&dev->power.lock);
 
+		/*
+		 * Resume suppliers if necessary.
+		 *
+		 * The device's runtime PM status cannot change until this
+		 * routine returns, so it is safe to read the status outside of
+		 * the lock.
+		 */
+		if (dev->power.runtime_status == RPM_RESUMING) {
+			idx = device_links_read_lock();
+
+			retval = rpm_get_suppliers(dev);
+			if (retval)
+				goto fail;
+
+			device_links_read_unlock(idx);
+		}
+	}
+
 	retval = cb(dev);
 
-	if (dev->power.irq_safe)
+	if (dev->power.irq_safe) {
 		spin_lock(&dev->power.lock);
-	else
+	} else {
+		/*
+		 * If the device is suspending and the callback has returned
+		 * success, drop the usage counters of the suppliers that have
+		 * been reference counted on its resume.
+		 *
+		 * Do that if resume fails too.
+		 */
+		if ((dev->power.runtime_status == RPM_SUSPENDING && !retval)
+		    || (dev->power.runtime_status == RPM_RESUMING && retval)) {
+			idx = device_links_read_lock();
+
+ fail:
+			rpm_put_suppliers(dev);
+
+			device_links_read_unlock(idx);
+		}
+
 		spin_lock_irq(&dev->power.lock);
+	}
 
 	return retval;
 }
@@ -1447,6 +1521,79 @@ void pm_runtime_remove(struct device *de
 }
 
 /**
+ * pm_runtime_clean_up_links - Prepare links to consumers for driver removal.
+ * @dev: Device whose driver is going to be removed.
+ *
+ * Check links from this device to any consumers and if any of them have active
+ * runtime PM references to the device, drop the usage counter of the device
+ * (once per link).
+ *
+ * Links with the DL_FLAG_STATELESS flag set are ignored.
+ *
+ * Since the device is guaranteed to be runtime-active at the point this is
+ * called, nothing else needs to be done here.
+ *
+ * Moreover, this is called after device_links_busy() has returned 'false', so
+ * the status of each link is guaranteed to be DL_STATE_SUPPLIER_UNBIND and
+ * therefore rpm_active can't be manipulated concurrently.
+ */
+void pm_runtime_clean_up_links(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links.consumers, s_node) {
+		if (link->flags & DL_FLAG_STATELESS)
+			continue;
+
+		if (link->rpm_active) {
+			pm_runtime_put_noidle(dev);
+			link->rpm_active = false;
+		}
+	}
+
+	device_links_read_unlock(idx);
+}
+
+/**
+ * pm_runtime_get_suppliers - Resume and reference-count supplier devices.
+ * @dev: Consumer device.
+ */
+void pm_runtime_get_suppliers(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links.suppliers, c_node)
+		if (link->flags & DL_FLAG_PM_RUNTIME)
+			pm_runtime_get_sync(link->supplier);
+
+	device_links_read_unlock(idx);
+}
+
+/**
+ * pm_runtime_put_suppliers - Drop references to supplier devices.
+ * @dev: Consumer device.
+ */
+void pm_runtime_put_suppliers(struct device *dev)
+{
+	struct device_link *link;
+	int idx;
+
+	idx = device_links_read_lock();
+
+	list_for_each_entry_rcu(link, &dev->links.suppliers, c_node)
+		if (link->flags & DL_FLAG_PM_RUNTIME)
+			pm_runtime_put(link->supplier);
+
+	device_links_read_unlock(idx);
+}
+
+/**
  * pm_runtime_force_suspend - Force a device into suspend state if needed.
  * @dev: Device to suspend.
  *
Index: linux-pm/include/linux/device.h
===================================================================
--- linux-pm.orig/include/linux/device.h
+++ linux-pm/include/linux/device.h
@@ -730,9 +730,13 @@ enum device_link_state {
  *
  * STATELESS: The core won't track the presence of supplier/consumer drivers.
  * AUTOREMOVE: Remove this link automatically on consumer driver unbind.
+ * PM_RUNTIME: If set, the runtime PM framework will use this link.
+ * RPM_ACTIVE: Run pm_runtime_get_sync() on the supplier during link creation.
  */
 #define DL_FLAG_STATELESS	BIT(0)
 #define DL_FLAG_AUTOREMOVE	BIT(1)
+#define DL_FLAG_PM_RUNTIME	BIT(2)
+#define DL_FLAG_RPM_ACTIVE	BIT(3)
 
 /**
  * struct device_link - Device link representation.
@@ -742,6 +746,7 @@ enum device_link_state {
  * @c_node: Hook to the consumer device's list of links to suppliers.
  * @status: The state of the link (with respect to the presence of drivers).
  * @flags: Link flags.
+ * @rpm_active: Whether or not the consumer device is runtime-PM-active.
  * @rcu_head: An RCU head to use for deferred execution of SRCU callbacks.
  */
 struct device_link {
@@ -751,6 +756,7 @@ struct device_link {
 	struct list_head c_node;
 	enum device_link_state status;
 	u32 flags;
+	bool rpm_active;
 #ifdef CONFIG_SRCU
 	struct rcu_head rcu_head;
 #endif
Index: linux-pm/include/linux/pm_runtime.h
===================================================================
--- linux-pm.orig/include/linux/pm_runtime.h
+++ linux-pm/include/linux/pm_runtime.h
@@ -55,6 +55,9 @@ extern unsigned long pm_runtime_autosusp
 extern void pm_runtime_update_max_time_suspended(struct device *dev,
 						 s64 delta_ns);
 extern void pm_runtime_set_memalloc_noio(struct device *dev, bool enable);
+extern void pm_runtime_clean_up_links(struct device *dev);
+extern void pm_runtime_get_suppliers(struct device *dev);
+extern void pm_runtime_put_suppliers(struct device *dev);
 
 static inline void pm_suspend_ignore_children(struct device *dev, bool enable)
 {
@@ -186,6 +189,9 @@ static inline unsigned long pm_runtime_a
 				struct device *dev) { return 0; }
 static inline void pm_runtime_set_memalloc_noio(struct device *dev,
 						bool enable){}
+static inline void pm_runtime_clean_up_links(struct device *dev) {}
+static inline void pm_runtime_get_suppliers(struct device *dev) {}
+static inline void pm_runtime_put_suppliers(struct device *dev) {}
 
 #endif /* !CONFIG_PM */
 

Re: [PATCH v6 4/5] PM / runtime: Use device links

[Lukas Wunner]

Hi Rafael,

spotted what looks like a bug in the device links runtime PM code:

When resuming a device, __rpm_callback() calls rpm_get_suppliers(dev):

> +			retval = rpm_get_suppliers(dev);
> +			if (retval)
> +				goto fail;


This will walk the list of suppliers and call pm_runtime_get_sync()
for each of them:

> +static int rpm_get_suppliers(struct device *dev)
> +{
> +	struct device_link *link;
> +
> +	list_for_each_entry_rcu(link, &dev->links.suppliers, c_node) {
> +		int retval;
[...]
> +		retval = pm_runtime_get_sync(link->supplier);
> +		if (retval < 0) {
> +			pm_runtime_put_noidle(link->supplier);
> +			return retval;


If pm_runtime_get_sync() failed, e.g. because runtime PM is disabled
on a supplier, the function will put the reference of the failed
supplier and return.

Back in __rpm_callback() we jump to the fail mark, where we call
rpm_put_suppliers().

> + fail:
> +			rpm_put_suppliers(dev);
> +
> +			device_links_read_unlock(idx);


This walks the list of suppliers and releases a ref for each of them:

> +static void rpm_put_suppliers(struct device *dev)
> +{
> +	struct device_link *link;
> +
> +	list_for_each_entry_rcu(link, &dev->links.suppliers, c_node)
> +		if (link->rpm_active &&
> +		    READ_ONCE(link->status) != DL_STATE_SUPPLIER_UNBIND) {
> +			pm_runtime_put(link->supplier);
> +			link->rpm_active = false;
> +		}
> +}


This looks wrong:  We've already put a ref on the failed supplier, so here
we're putting another one.  And if there are further suppliers in the list
following the failed one, we'll decrement their refcount even though we've
never incremented it.

Thanks,

Lukas

Re: [PATCH v6 4/5] PM / runtime: Use device links

[Rafael J. Wysocki]

On Sun, Dec 18, 2016 at 3:01 PM, Lukas Wunner <lukas@wunner.de> wrote:
> Hi Rafael,
>
> spotted what looks like a bug in the device links runtime PM code:
>
> When resuming a device, __rpm_callback() calls rpm_get_suppliers(dev):
>
>> +                     retval = rpm_get_suppliers(dev);
>> +                     if (retval)
>> +                             goto fail;
>
>
> This will walk the list of suppliers and call pm_runtime_get_sync()
> for each of them:
>
>> +static int rpm_get_suppliers(struct device *dev)
>> +{
>> +     struct device_link *link;
>> +
>> +     list_for_each_entry_rcu(link, &dev->links.suppliers, c_node) {
>> +             int retval;
> [...]
>> +             retval = pm_runtime_get_sync(link->supplier);
>> +             if (retval < 0) {
>> +                     pm_runtime_put_noidle(link->supplier);
>> +                     return retval;
>
>
> If pm_runtime_get_sync() failed, e.g. because runtime PM is disabled
> on a supplier, the function will put the reference of the failed
> supplier and return.
>
> Back in __rpm_callback() we jump to the fail mark, where we call
> rpm_put_suppliers().
>
>> + fail:
>> +                     rpm_put_suppliers(dev);
>> +
>> +                     device_links_read_unlock(idx);
>
>
> This walks the list of suppliers and releases a ref for each of them:
>
>> +static void rpm_put_suppliers(struct device *dev)
>> +{
>> +     struct device_link *link;
>> +
>> +     list_for_each_entry_rcu(link, &dev->links.suppliers, c_node)
>> +             if (link->rpm_active &&
>> +                 READ_ONCE(link->status) != DL_STATE_SUPPLIER_UNBIND) {
>> +                     pm_runtime_put(link->supplier);
>> +                     link->rpm_active = false;
>> +             }
>> +}
>
>
> This looks wrong:  We've already put a ref on the failed supplier, so here
> we're putting another one.

Are we?  I would think link->rpm_active would be false for the failed
one, wouldn't it?

>  And if there are further suppliers in the list
> following the failed one, we'll decrement their refcount even though we've
> never incremented it.

I'm not following you here, sorry.

Thanks,
Rafael

Re: [PATCH v6 4/5] PM / runtime: Use device links

[Lukas Wunner]

On Sun, Dec 18, 2016 at 04:53:26PM +0100, Rafael J. Wysocki wrote:
> On Sun, Dec 18, 2016 at 3:01 PM, Lukas Wunner <lukas@wunner.de> wrote:
> > Hi Rafael,
> >
> > spotted what looks like a bug in the device links runtime PM code:
> >
> > When resuming a device, __rpm_callback() calls rpm_get_suppliers(dev):
> >
> >> +                     retval = rpm_get_suppliers(dev);
> >> +                     if (retval)
> >> +                             goto fail;
> >
> >
> > This will walk the list of suppliers and call pm_runtime_get_sync()
> > for each of them:
> >
> >> +static int rpm_get_suppliers(struct device *dev)
> >> +{
> >> +     struct device_link *link;
> >> +
> >> +     list_for_each_entry_rcu(link, &dev->links.suppliers, c_node) {
> >> +             int retval;
> > [...]
> >> +             retval = pm_runtime_get_sync(link->supplier);
> >> +             if (retval < 0) {
> >> +                     pm_runtime_put_noidle(link->supplier);
> >> +                     return retval;
> >
> >
> > If pm_runtime_get_sync() failed, e.g. because runtime PM is disabled
> > on a supplier, the function will put the reference of the failed
> > supplier and return.
> >
> > Back in __rpm_callback() we jump to the fail mark, where we call
> > rpm_put_suppliers().
> >
> >> + fail:
> >> +                     rpm_put_suppliers(dev);
> >> +
> >> +                     device_links_read_unlock(idx);
> >
> >
> > This walks the list of suppliers and releases a ref for each of them:
> >
> >> +static void rpm_put_suppliers(struct device *dev)
> >> +{
> >> +     struct device_link *link;
> >> +
> >> +     list_for_each_entry_rcu(link, &dev->links.suppliers, c_node)
> >> +             if (link->rpm_active &&
> >> +                 READ_ONCE(link->status) != DL_STATE_SUPPLIER_UNBIND) {
> >> +                     pm_runtime_put(link->supplier);
> >> +                     link->rpm_active = false;
> >> +             }
> >> +}
> >
> >
> > This looks wrong:  We've already put a ref on the failed supplier, so here
> > we're putting another one.
> 
> Are we?  I would think link->rpm_active would be false for the failed
> one, wouldn't it?

Ah, so link->rpm_active means the consumer is holding a ref on the supplier.
Missed that, sorry for the false alarm and thanks for the clarification.

Lukas

Re: [PATCH v6 4/5] PM / runtime: Use device links

[Rafael J. Wysocki]

On Sun, Dec 18, 2016 at 5:37 PM, Lukas Wunner <lukas@wunner.de> wrote:
> On Sun, Dec 18, 2016 at 04:53:26PM +0100, Rafael J. Wysocki wrote:
>> On Sun, Dec 18, 2016 at 3:01 PM, Lukas Wunner <lukas@wunner.de> wrote:
>> > Hi Rafael,
>> >
>> > spotted what looks like a bug in the device links runtime PM code:
>> >
>> > When resuming a device, __rpm_callback() calls rpm_get_suppliers(dev):
>> >
>> >> +                     retval = rpm_get_suppliers(dev);
>> >> +                     if (retval)
>> >> +                             goto fail;
>> >
>> >
>> > This will walk the list of suppliers and call pm_runtime_get_sync()
>> > for each of them:
>> >
>> >> +static int rpm_get_suppliers(struct device *dev)
>> >> +{
>> >> +     struct device_link *link;
>> >> +
>> >> +     list_for_each_entry_rcu(link, &dev->links.suppliers, c_node) {
>> >> +             int retval;
>> > [...]
>> >> +             retval = pm_runtime_get_sync(link->supplier);
>> >> +             if (retval < 0) {
>> >> +                     pm_runtime_put_noidle(link->supplier);
>> >> +                     return retval;
>> >
>> >
>> > If pm_runtime_get_sync() failed, e.g. because runtime PM is disabled
>> > on a supplier, the function will put the reference of the failed
>> > supplier and return.
>> >
>> > Back in __rpm_callback() we jump to the fail mark, where we call
>> > rpm_put_suppliers().
>> >
>> >> + fail:
>> >> +                     rpm_put_suppliers(dev);
>> >> +
>> >> +                     device_links_read_unlock(idx);
>> >
>> >
>> > This walks the list of suppliers and releases a ref for each of them:
>> >
>> >> +static void rpm_put_suppliers(struct device *dev)
>> >> +{
>> >> +     struct device_link *link;
>> >> +
>> >> +     list_for_each_entry_rcu(link, &dev->links.suppliers, c_node)
>> >> +             if (link->rpm_active &&
>> >> +                 READ_ONCE(link->status) != DL_STATE_SUPPLIER_UNBIND) {
>> >> +                     pm_runtime_put(link->supplier);
>> >> +                     link->rpm_active = false;
>> >> +             }
>> >> +}
>> >
>> >
>> > This looks wrong:  We've already put a ref on the failed supplier, so here
>> > we're putting another one.
>>
>> Are we?  I would think link->rpm_active would be false for the failed
>> one, wouldn't it?
>
> Ah, so link->rpm_active means the consumer is holding a ref on the supplier.

Yes, that's the idea. :-)

> Missed that, sorry for the false alarm and thanks for the clarification.

No problem.

Thanks,
Rafael

[Resend][PATCH v6 5/5] PM / runtime: Optimize the use of device links

[Rafael J. Wysocki]

From: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Subject: 

If the device has no links to suppliers that should be used for
runtime PM (links with DEVICE_LINK_PM_RUNTIME set), there is no
reason to walk the list of suppliers for that device during
runtime suspend and resume.

Add a simple mechanism to detect that case and possibly avoid the
extra unnecessary overhead.

Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/base/core.c          |   20 +++++++++++++-------
 drivers/base/power/runtime.c |   23 ++++++++++++++++++++---
 include/linux/pm.h           |    1 +
 include/linux/pm_runtime.h   |    4 ++++
 4 files changed, 38 insertions(+), 10 deletions(-)

Index: linux-pm/drivers/base/core.c
===================================================================
--- linux-pm.orig/drivers/base/core.c
+++ linux-pm/drivers/base/core.c
@@ -205,14 +205,17 @@ struct device_link *device_link_add(stru
 	if (!link)
 		goto out;
 
-	if ((flags & DL_FLAG_PM_RUNTIME) && (flags & DL_FLAG_RPM_ACTIVE)) {
-		if (pm_runtime_get_sync(supplier) < 0) {
-			pm_runtime_put_noidle(supplier);
-			kfree(link);
-			link = NULL;
-			goto out;
+	if (flags & DL_FLAG_PM_RUNTIME) {
+		if (flags & DL_FLAG_RPM_ACTIVE) {
+			if (pm_runtime_get_sync(supplier) < 0) {
+				pm_runtime_put_noidle(supplier);
+				kfree(link);
+				link = NULL;
+				goto out;
+			}
+			link->rpm_active = true;
 		}
-		link->rpm_active = true;
+		pm_runtime_new_link(consumer);
 	}
 	get_device(supplier);
 	link->supplier = supplier;
@@ -296,6 +299,9 @@ static void __device_link_del(struct dev
 	dev_info(link->consumer, "Dropping the link to %s\n",
 		 dev_name(link->supplier));
 
+	if (link->flags & DL_FLAG_PM_RUNTIME)
+		pm_runtime_drop_link(link->consumer);
+
 	list_del_rcu(&link->s_node);
 	list_del_rcu(&link->c_node);
 	call_srcu(&device_links_srcu, &link->rcu_head, __device_link_free_srcu);
Index: linux-pm/drivers/base/power/runtime.c
===================================================================
--- linux-pm.orig/drivers/base/power/runtime.c
+++ linux-pm/drivers/base/power/runtime.c
@@ -305,6 +305,7 @@ static int __rpm_callback(int (*cb)(stru
 	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
 	int retval, idx;
+	bool use_links = dev->power.links_count > 0;
 
 	if (dev->power.irq_safe) {
 		spin_unlock(&dev->power.lock);
@@ -318,7 +319,7 @@ static int __rpm_callback(int (*cb)(stru
 		 * routine returns, so it is safe to read the status outside of
 		 * the lock.
 		 */
-		if (dev->power.runtime_status == RPM_RESUMING) {
+		if (use_links && dev->power.runtime_status == RPM_RESUMING) {
 			idx = device_links_read_lock();
 
 			retval = rpm_get_suppliers(dev);
@@ -341,8 +342,9 @@ static int __rpm_callback(int (*cb)(stru
 		 *
 		 * Do that if resume fails too.
 		 */
-		if ((dev->power.runtime_status == RPM_SUSPENDING && !retval)
-		    || (dev->power.runtime_status == RPM_RESUMING && retval)) {
+		if (use_links
+		    && ((dev->power.runtime_status == RPM_SUSPENDING && !retval)
+		    || (dev->power.runtime_status == RPM_RESUMING && retval))) {
 			idx = device_links_read_lock();
 
  fail:
@@ -1593,6 +1595,21 @@ void pm_runtime_put_suppliers(struct dev
 	device_links_read_unlock(idx);
 }
 
+void pm_runtime_new_link(struct device *dev)
+{
+	spin_lock_irq(&dev->power.lock);
+	dev->power.links_count++;
+	spin_unlock_irq(&dev->power.lock);
+}
+
+void pm_runtime_drop_link(struct device *dev)
+{
+	spin_lock_irq(&dev->power.lock);
+	WARN_ON(dev->power.links_count == 0);
+	dev->power.links_count--;
+	spin_unlock_irq(&dev->power.lock);
+}
+
 /**
  * pm_runtime_force_suspend - Force a device into suspend state if needed.
  * @dev: Device to suspend.
Index: linux-pm/include/linux/pm.h
===================================================================
--- linux-pm.orig/include/linux/pm.h
+++ linux-pm/include/linux/pm.h
@@ -597,6 +597,7 @@ struct dev_pm_info {
 	unsigned int		use_autosuspend:1;
 	unsigned int		timer_autosuspends:1;
 	unsigned int		memalloc_noio:1;
+	unsigned int		links_count;
 	enum rpm_request	request;
 	enum rpm_status		runtime_status;
 	int			runtime_error;
Index: linux-pm/include/linux/pm_runtime.h
===================================================================
--- linux-pm.orig/include/linux/pm_runtime.h
+++ linux-pm/include/linux/pm_runtime.h
@@ -58,6 +58,8 @@ extern void pm_runtime_set_memalloc_noio
 extern void pm_runtime_clean_up_links(struct device *dev);
 extern void pm_runtime_get_suppliers(struct device *dev);
 extern void pm_runtime_put_suppliers(struct device *dev);
+extern void pm_runtime_new_link(struct device *dev);
+extern void pm_runtime_drop_link(struct device *dev);
 
 static inline void pm_suspend_ignore_children(struct device *dev, bool enable)
 {
@@ -192,6 +194,8 @@ static inline void pm_runtime_set_memall
 static inline void pm_runtime_clean_up_links(struct device *dev) {}
 static inline void pm_runtime_get_suppliers(struct device *dev) {}
 static inline void pm_runtime_put_suppliers(struct device *dev) {}
+static inline void pm_runtime_new_link(struct device *dev) {}
+static inline void pm_runtime_drop_link(struct device *dev) {}
 
 #endif /* !CONFIG_PM */
 

Re: [PATCH v6 0/5] Functional dependencies between devices

[Rafael J. Wysocki]

On Sunday, October 30, 2016 05:22:13 PM Rafael J. Wysocki wrote:
> Hi,
> 
> Let me quote from the previous intro messages for this series first:
> 
> > > Time for another update. :-)
> > > 
> > > Fewer changes this time, mostly to address issues found by Lukas and
> > > Marek.
> > > 
> > > The most significant one is to make device_link_add() cope with the case
> > > when
> > > the consumer device has not been registered yet when it is called.  The
> > > supplier device still is required to be registered and the function will
> > > return NULL if that is not the case.
> > > 
> > > Another significant change is in patch [4/5] that now makes the core apply
> > > pm_runtime_get_sync()/pm_runtime_put() to supplier devices around the
> > > probing of a consumer one (in analogy with the parent).
> > 
> > One more update after some conversations during LinuxCon Europe.
> > 
> > The main point was to make it possible for device_link_add() to figure out
> > the initial state of the link instead of expecting the caller to provide it
> > which might not be reliable enough in general.
> > 
> > In this version device_link_add() takes three arguments, the supplier and
> > consumer pointers and flags and it sets the correct initial state of the
> > link automatically (unless invoked with the "stateless" flag, of course).
> > The cost is one additional field in struct device (I moved all of the
> > links-related fields in struct device to a separate sub-structure while at
> > it) to track the "driver presence status" of the device (to be used by
> > device_link_add()).
> > 
> > In addition to that, the links list walks in the core.c and dd.c code are
> > under the device links mutex now, so the iternal link spinlock is not needed
> > any more and I have renamed symbols to distinguish between flags, link
> > states and device "driver presence statuses".
> 
> The most significant change in this revision with respect to the previous one is
> related to the fact that SRCU is not available on some architectures, so the
> code falls back to using an RW semaphore for synchronization if SRCU is not
> there.  Fortunately, the code changes needed for that turned out to be quite
> straightforward and confined to the second patch.
> 
> Apart from this, the flags are defined using BIT(x) now (instead of open coding
> the latter in the flag definitions).
> 
> Updated is mostly patch [2/5].  Patches [1,3,5/5] have not changed (except for
> trivial rebasing) and patch [4/5] needed to be refreshed on top of the modified
> [2/5].
> 
> FWIW, I've run the series through 0-day which has not reported any problems
> with it.

BTW, the series is available from the device-links-test branch in my tree:

git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git device-links-test

in case someone wants to try it out.

Thanks,
Rafael

Re: [PATCH v6 0/5] Functional dependencies between devices

[Greg Kroah-Hartman]

On Sun, Oct 30, 2016 at 05:22:13PM +0100, Rafael J. Wysocki wrote:
> Hi,
> 
> Let me quote from the previous intro messages for this series first:
> 
> > > Time for another update. :-)
> > > 
> > > Fewer changes this time, mostly to address issues found by Lukas and
> > > Marek.
> > > 
> > > The most significant one is to make device_link_add() cope with the case
> > > when
> > > the consumer device has not been registered yet when it is called.  The
> > > supplier device still is required to be registered and the function will
> > > return NULL if that is not the case.
> > > 
> > > Another significant change is in patch [4/5] that now makes the core apply
> > > pm_runtime_get_sync()/pm_runtime_put() to supplier devices around the
> > > probing of a consumer one (in analogy with the parent).
> > 
> > One more update after some conversations during LinuxCon Europe.
> > 
> > The main point was to make it possible for device_link_add() to figure out
> > the initial state of the link instead of expecting the caller to provide it
> > which might not be reliable enough in general.
> > 
> > In this version device_link_add() takes three arguments, the supplier and
> > consumer pointers and flags and it sets the correct initial state of the
> > link automatically (unless invoked with the "stateless" flag, of course).
> > The cost is one additional field in struct device (I moved all of the
> > links-related fields in struct device to a separate sub-structure while at
> > it) to track the "driver presence status" of the device (to be used by
> > device_link_add()).
> > 
> > In addition to that, the links list walks in the core.c and dd.c code are
> > under the device links mutex now, so the iternal link spinlock is not needed
> > any more and I have renamed symbols to distinguish between flags, link
> > states and device "driver presence statuses".
> 
> The most significant change in this revision with respect to the previous one is
> related to the fact that SRCU is not available on some architectures, so the
> code falls back to using an RW semaphore for synchronization if SRCU is not
> there.  Fortunately, the code changes needed for that turned out to be quite
> straightforward and confined to the second patch.
> 
> Apart from this, the flags are defined using BIT(x) now (instead of open coding
> the latter in the flag definitions).
> 
> Updated is mostly patch [2/5].  Patches [1,3,5/5] have not changed (except for
> trivial rebasing) and patch [4/5] needed to be refreshed on top of the modified
> [2/5].
> 
> FWIW, I've run the series through 0-day which has not reported any problems
> with it.

Great, they are now applied to my tree, thanks again for doing this
work.

greg k-h

Re: [PATCH v6 0/5] Functional dependencies between devices

[Rafael J. Wysocki]

On Monday, October 31, 2016 11:47:03 AM Greg Kroah-Hartman wrote:
> On Sun, Oct 30, 2016 at 05:22:13PM +0100, Rafael J. Wysocki wrote:
> > Hi,
> > 
> > Let me quote from the previous intro messages for this series first:
> > 
> > > > Time for another update. :-)
> > > > 
> > > > Fewer changes this time, mostly to address issues found by Lukas and
> > > > Marek.
> > > > 
> > > > The most significant one is to make device_link_add() cope with the case
> > > > when
> > > > the consumer device has not been registered yet when it is called.  The
> > > > supplier device still is required to be registered and the function will
> > > > return NULL if that is not the case.
> > > > 
> > > > Another significant change is in patch [4/5] that now makes the core apply
> > > > pm_runtime_get_sync()/pm_runtime_put() to supplier devices around the
> > > > probing of a consumer one (in analogy with the parent).
> > > 
> > > One more update after some conversations during LinuxCon Europe.
> > > 
> > > The main point was to make it possible for device_link_add() to figure out
> > > the initial state of the link instead of expecting the caller to provide it
> > > which might not be reliable enough in general.
> > > 
> > > In this version device_link_add() takes three arguments, the supplier and
> > > consumer pointers and flags and it sets the correct initial state of the
> > > link automatically (unless invoked with the "stateless" flag, of course).
> > > The cost is one additional field in struct device (I moved all of the
> > > links-related fields in struct device to a separate sub-structure while at
> > > it) to track the "driver presence status" of the device (to be used by
> > > device_link_add()).
> > > 
> > > In addition to that, the links list walks in the core.c and dd.c code are
> > > under the device links mutex now, so the iternal link spinlock is not needed
> > > any more and I have renamed symbols to distinguish between flags, link
> > > states and device "driver presence statuses".
> > 
> > The most significant change in this revision with respect to the previous one is
> > related to the fact that SRCU is not available on some architectures, so the
> > code falls back to using an RW semaphore for synchronization if SRCU is not
> > there.  Fortunately, the code changes needed for that turned out to be quite
> > straightforward and confined to the second patch.
> > 
> > Apart from this, the flags are defined using BIT(x) now (instead of open coding
> > the latter in the flag definitions).
> > 
> > Updated is mostly patch [2/5].  Patches [1,3,5/5] have not changed (except for
> > trivial rebasing) and patch [4/5] needed to be refreshed on top of the modified
> > [2/5].
> > 
> > FWIW, I've run the series through 0-day which has not reported any problems
> > with it.
> 
> Great, they are now applied to my tree, thanks again for doing this
> work.

Thanks!

Re: [PATCH v6 0/5] Functional dependencies between devices

[Marek Szyprowski]

Hi Greg,


On 2016-10-31 18:47, Greg Kroah-Hartman wrote:
> On Sun, Oct 30, 2016 at 05:22:13PM +0100, Rafael J. Wysocki wrote:
>> Let me quote from the previous intro messages for this series first:
>>
>>>> Time for another update. :-)
>>>>
>>>> Fewer changes this time, mostly to address issues found by Lukas and
>>>> Marek.
>>>>
>>>> The most significant one is to make device_link_add() cope with the case
>>>> when
>>>> the consumer device has not been registered yet when it is called.  The
>>>> supplier device still is required to be registered and the function will
>>>> return NULL if that is not the case.
>>>>
>>>> Another significant change is in patch [4/5] that now makes the core apply
>>>> pm_runtime_get_sync()/pm_runtime_put() to supplier devices around the
>>>> probing of a consumer one (in analogy with the parent).
>>> One more update after some conversations during LinuxCon Europe.
>>>
>>> The main point was to make it possible for device_link_add() to figure out
>>> the initial state of the link instead of expecting the caller to provide it
>>> which might not be reliable enough in general.
>>>
>>> In this version device_link_add() takes three arguments, the supplier and
>>> consumer pointers and flags and it sets the correct initial state of the
>>> link automatically (unless invoked with the "stateless" flag, of course).
>>> The cost is one additional field in struct device (I moved all of the
>>> links-related fields in struct device to a separate sub-structure while at
>>> it) to track the "driver presence status" of the device (to be used by
>>> device_link_add()).
>>>
>>> In addition to that, the links list walks in the core.c and dd.c code are
>>> under the device links mutex now, so the iternal link spinlock is not needed
>>> any more and I have renamed symbols to distinguish between flags, link
>>> states and device "driver presence statuses".
>> The most significant change in this revision with respect to the previous one is
>> related to the fact that SRCU is not available on some architectures, so the
>> code falls back to using an RW semaphore for synchronization if SRCU is not
>> there.  Fortunately, the code changes needed for that turned out to be quite
>> straightforward and confined to the second patch.
>>
>> Apart from this, the flags are defined using BIT(x) now (instead of open coding
>> the latter in the flag definitions).
>>
>> Updated is mostly patch [2/5].  Patches [1,3,5/5] have not changed (except for
>> trivial rebasing) and patch [4/5] needed to be refreshed on top of the modified
>> [2/5].
>>
>> FWIW, I've run the series through 0-day which has not reported any problems
>> with it.
> Great, they are now applied to my tree, thanks again for doing this
> work.

Thanks for merging those patches! Could you provide a stable tag with 
them, so I can
ask Joerg to merge my Exynos IOMMU PM patches on top of it via IOMMU tree?

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [PATCH v6 0/5] Functional dependencies between devices

[Greg Kroah-Hartman]

On Wed, Nov 02, 2016 at 08:58:38AM +0100, Marek Szyprowski wrote:
> Hi Greg,
> 
> 
> On 2016-10-31 18:47, Greg Kroah-Hartman wrote:
> > On Sun, Oct 30, 2016 at 05:22:13PM +0100, Rafael J. Wysocki wrote:
> > > Let me quote from the previous intro messages for this series first:
> > > 
> > > > > Time for another update. :-)
> > > > > 
> > > > > Fewer changes this time, mostly to address issues found by Lukas and
> > > > > Marek.
> > > > > 
> > > > > The most significant one is to make device_link_add() cope with the case
> > > > > when
> > > > > the consumer device has not been registered yet when it is called.  The
> > > > > supplier device still is required to be registered and the function will
> > > > > return NULL if that is not the case.
> > > > > 
> > > > > Another significant change is in patch [4/5] that now makes the core apply
> > > > > pm_runtime_get_sync()/pm_runtime_put() to supplier devices around the
> > > > > probing of a consumer one (in analogy with the parent).
> > > > One more update after some conversations during LinuxCon Europe.
> > > > 
> > > > The main point was to make it possible for device_link_add() to figure out
> > > > the initial state of the link instead of expecting the caller to provide it
> > > > which might not be reliable enough in general.
> > > > 
> > > > In this version device_link_add() takes three arguments, the supplier and
> > > > consumer pointers and flags and it sets the correct initial state of the
> > > > link automatically (unless invoked with the "stateless" flag, of course).
> > > > The cost is one additional field in struct device (I moved all of the
> > > > links-related fields in struct device to a separate sub-structure while at
> > > > it) to track the "driver presence status" of the device (to be used by
> > > > device_link_add()).
> > > > 
> > > > In addition to that, the links list walks in the core.c and dd.c code are
> > > > under the device links mutex now, so the iternal link spinlock is not needed
> > > > any more and I have renamed symbols to distinguish between flags, link
> > > > states and device "driver presence statuses".
> > > The most significant change in this revision with respect to the previous one is
> > > related to the fact that SRCU is not available on some architectures, so the
> > > code falls back to using an RW semaphore for synchronization if SRCU is not
> > > there.  Fortunately, the code changes needed for that turned out to be quite
> > > straightforward and confined to the second patch.
> > > 
> > > Apart from this, the flags are defined using BIT(x) now (instead of open coding
> > > the latter in the flag definitions).
> > > 
> > > Updated is mostly patch [2/5].  Patches [1,3,5/5] have not changed (except for
> > > trivial rebasing) and patch [4/5] needed to be refreshed on top of the modified
> > > [2/5].
> > > 
> > > FWIW, I've run the series through 0-day which has not reported any problems
> > > with it.
> > Great, they are now applied to my tree, thanks again for doing this
> > work.
> 
> Thanks for merging those patches! Could you provide a stable tag with them,
> so I can
> ask Joerg to merge my Exynos IOMMU PM patches on top of it via IOMMU tree?

My trees do not get rebased so you can pull from it directly right now,
or if you really need a signed tag, I can make one up, but it will not
be until Monday that I can do that.

thanks,

greg k-h

Re: [PATCH v6 0/5] Functional dependencies between devices

[Luis R. Rodriguez]

On Wed, Nov 02, 2016 at 08:58:38AM +0100, Marek Szyprowski wrote:
> Hi Greg,
> 
> 
> On 2016-10-31 18:47, Greg Kroah-Hartman wrote:
> > On Sun, Oct 30, 2016 at 05:22:13PM +0100, Rafael J. Wysocki wrote:
> > > Let me quote from the previous intro messages for this series first:
> > > 
> > > > > Time for another update. :-)
> > > > > 
> > > > > Fewer changes this time, mostly to address issues found by Lukas and
> > > > > Marek.
> > > > > 
> > > > > The most significant one is to make device_link_add() cope with the case
> > > > > when
> > > > > the consumer device has not been registered yet when it is called.  The
> > > > > supplier device still is required to be registered and the function will
> > > > > return NULL if that is not the case.
> > > > > 
> > > > > Another significant change is in patch [4/5] that now makes the core apply
> > > > > pm_runtime_get_sync()/pm_runtime_put() to supplier devices around the
> > > > > probing of a consumer one (in analogy with the parent).
> > > > One more update after some conversations during LinuxCon Europe.
> > > > 
> > > > The main point was to make it possible for device_link_add() to figure out
> > > > the initial state of the link instead of expecting the caller to provide it
> > > > which might not be reliable enough in general.
> > > > 
> > > > In this version device_link_add() takes three arguments, the supplier and
> > > > consumer pointers and flags and it sets the correct initial state of the
> > > > link automatically (unless invoked with the "stateless" flag, of course).
> > > > The cost is one additional field in struct device (I moved all of the
> > > > links-related fields in struct device to a separate sub-structure while at
> > > > it) to track the "driver presence status" of the device (to be used by
> > > > device_link_add()).
> > > > 
> > > > In addition to that, the links list walks in the core.c and dd.c code are
> > > > under the device links mutex now, so the iternal link spinlock is not needed
> > > > any more and I have renamed symbols to distinguish between flags, link
> > > > states and device "driver presence statuses".
> > > The most significant change in this revision with respect to the previous one is
> > > related to the fact that SRCU is not available on some architectures, so the
> > > code falls back to using an RW semaphore for synchronization if SRCU is not
> > > there.  Fortunately, the code changes needed for that turned out to be quite
> > > straightforward and confined to the second patch.
> > > 
> > > Apart from this, the flags are defined using BIT(x) now (instead of open coding
> > > the latter in the flag definitions).
> > > 
> > > Updated is mostly patch [2/5].  Patches [1,3,5/5] have not changed (except for
> > > trivial rebasing) and patch [4/5] needed to be refreshed on top of the modified
> > > [2/5].
> > > 
> > > FWIW, I've run the series through 0-day which has not reported any problems
> > > with it.
> > Great, they are now applied to my tree, thanks again for doing this
> > work.
> 
> Thanks for merging those patches! Could you provide a stable tag with them,
> so I can
> ask Joerg to merge my Exynos IOMMU PM patches on top of it via IOMMU tree?

You want these patches to be merged into stable?! This is a whole new set of
functionality, the patches in no way describe any *fixes* or critical issues,
why are you saying this is needed? What makes you believe this is a stable
candidate?

 Luis

Re: [PATCH v6 0/5] Functional dependencies between devices

[Marek Szyprowski]

Hi Luis,


On 2016-11-07 22:15, Luis R. Rodriguez wrote:
> On Wed, Nov 02, 2016 at 08:58:38AM +0100, Marek Szyprowski wrote:
>> On 2016-10-31 18:47, Greg Kroah-Hartman wrote:
>>> On Sun, Oct 30, 2016 at 05:22:13PM +0100, Rafael J. Wysocki wrote:
>>>> Let me quote from the previous intro messages for this series first:
>>>>
>>>>>> Time for another update. :-)
>>>>>>
>>>>>> Fewer changes this time, mostly to address issues found by Lukas and
>>>>>> Marek.
>>>>>>
>>>>>> The most significant one is to make device_link_add() cope with the case
>>>>>> when
>>>>>> the consumer device has not been registered yet when it is called.  The
>>>>>> supplier device still is required to be registered and the function will
>>>>>> return NULL if that is not the case.
>>>>>>
>>>>>> Another significant change is in patch [4/5] that now makes the core apply
>>>>>> pm_runtime_get_sync()/pm_runtime_put() to supplier devices around the
>>>>>> probing of a consumer one (in analogy with the parent).
>>>>> One more update after some conversations during LinuxCon Europe.
>>>>>
>>>>> The main point was to make it possible for device_link_add() to figure out
>>>>> the initial state of the link instead of expecting the caller to provide it
>>>>> which might not be reliable enough in general.
>>>>>
>>>>> In this version device_link_add() takes three arguments, the supplier and
>>>>> consumer pointers and flags and it sets the correct initial state of the
>>>>> link automatically (unless invoked with the "stateless" flag, of course).
>>>>> The cost is one additional field in struct device (I moved all of the
>>>>> links-related fields in struct device to a separate sub-structure while at
>>>>> it) to track the "driver presence status" of the device (to be used by
>>>>> device_link_add()).
>>>>>
>>>>> In addition to that, the links list walks in the core.c and dd.c code are
>>>>> under the device links mutex now, so the iternal link spinlock is not needed
>>>>> any more and I have renamed symbols to distinguish between flags, link
>>>>> states and device "driver presence statuses".
>>>> The most significant change in this revision with respect to the previous one is
>>>> related to the fact that SRCU is not available on some architectures, so the
>>>> code falls back to using an RW semaphore for synchronization if SRCU is not
>>>> there.  Fortunately, the code changes needed for that turned out to be quite
>>>> straightforward and confined to the second patch.
>>>>
>>>> Apart from this, the flags are defined using BIT(x) now (instead of open coding
>>>> the latter in the flag definitions).
>>>>
>>>> Updated is mostly patch [2/5].  Patches [1,3,5/5] have not changed (except for
>>>> trivial rebasing) and patch [4/5] needed to be refreshed on top of the modified
>>>> [2/5].
>>>>
>>>> FWIW, I've run the series through 0-day which has not reported any problems
>>>> with it.
>>> Great, they are now applied to my tree, thanks again for doing this
>>> work.
>> Thanks for merging those patches! Could you provide a stable tag with them,
>> so I can
>> ask Joerg to merge my Exynos IOMMU PM patches on top of it via IOMMU tree?
> You want these patches to be merged into stable?! This is a whole new set of
> functionality, the patches in no way describe any *fixes* or critical issues,
> why are you saying this is needed? What makes you believe this is a stable
> candidate?

I don't want to merge those patches to stale kernel release. By 'stable 
tag' I just
meant something that can be pulled by Joerg to have a base for my Exynos 
IOMMU
patches.

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [PATCH v6 0/5] Functional dependencies between devices

[Luis R. Rodriguez]

On Tue, Nov 08, 2016 at 07:36:45AM +0100, Marek Szyprowski wrote:
> Hi Luis,
> 
> 
> On 2016-11-07 22:15, Luis R. Rodriguez wrote:
> > On Wed, Nov 02, 2016 at 08:58:38AM +0100, Marek Szyprowski wrote:
> > > On 2016-10-31 18:47, Greg Kroah-Hartman wrote:
> > > > On Sun, Oct 30, 2016 at 05:22:13PM +0100, Rafael J. Wysocki wrote:
> > > > > Let me quote from the previous intro messages for this series first:
> > > > > 
> > > > > > > Time for another update. :-)
> > > > > > > 
> > > > > > > Fewer changes this time, mostly to address issues found by Lukas and
> > > > > > > Marek.
> > > > > > > 
> > > > > > > The most significant one is to make device_link_add() cope with the case
> > > > > > > when
> > > > > > > the consumer device has not been registered yet when it is called.  The
> > > > > > > supplier device still is required to be registered and the function will
> > > > > > > return NULL if that is not the case.
> > > > > > > 
> > > > > > > Another significant change is in patch [4/5] that now makes the core apply
> > > > > > > pm_runtime_get_sync()/pm_runtime_put() to supplier devices around the
> > > > > > > probing of a consumer one (in analogy with the parent).
> > > > > > One more update after some conversations during LinuxCon Europe.
> > > > > > 
> > > > > > The main point was to make it possible for device_link_add() to figure out
> > > > > > the initial state of the link instead of expecting the caller to provide it
> > > > > > which might not be reliable enough in general.
> > > > > > 
> > > > > > In this version device_link_add() takes three arguments, the supplier and
> > > > > > consumer pointers and flags and it sets the correct initial state of the
> > > > > > link automatically (unless invoked with the "stateless" flag, of course).
> > > > > > The cost is one additional field in struct device (I moved all of the
> > > > > > links-related fields in struct device to a separate sub-structure while at
> > > > > > it) to track the "driver presence status" of the device (to be used by
> > > > > > device_link_add()).
> > > > > > 
> > > > > > In addition to that, the links list walks in the core.c and dd.c code are
> > > > > > under the device links mutex now, so the iternal link spinlock is not needed
> > > > > > any more and I have renamed symbols to distinguish between flags, link
> > > > > > states and device "driver presence statuses".
> > > > > The most significant change in this revision with respect to the previous one is
> > > > > related to the fact that SRCU is not available on some architectures, so the
> > > > > code falls back to using an RW semaphore for synchronization if SRCU is not
> > > > > there.  Fortunately, the code changes needed for that turned out to be quite
> > > > > straightforward and confined to the second patch.
> > > > > 
> > > > > Apart from this, the flags are defined using BIT(x) now (instead of open coding
> > > > > the latter in the flag definitions).
> > > > > 
> > > > > Updated is mostly patch [2/5].  Patches [1,3,5/5] have not changed (except for
> > > > > trivial rebasing) and patch [4/5] needed to be refreshed on top of the modified
> > > > > [2/5].
> > > > > 
> > > > > FWIW, I've run the series through 0-day which has not reported any problems
> > > > > with it.
> > > > Great, they are now applied to my tree, thanks again for doing this
> > > > work.
> > > Thanks for merging those patches! Could you provide a stable tag with them,
> > > so I can
> > > ask Joerg to merge my Exynos IOMMU PM patches on top of it via IOMMU tree?
> > You want these patches to be merged into stable?! This is a whole new set of
> > functionality, the patches in no way describe any *fixes* or critical issues,
> > why are you saying this is needed? What makes you believe this is a stable
> > candidate?
> 
> I don't want to merge those patches to stale kernel release. By 'stable tag'
> I just meant something that can be pulled by Joerg to have a base for my
> Exynos IOMMU patches.

Phew! Thanks for the clarification!

  Luis