[PATCH v5 0/2] Synchronize DT overlay removal with devlink removals

[PATCH v5 0/2] Synchronize DT overlay removal with devlink removals

[Herve Codina]

Hi,

In the following sequence:
  of_platform_depopulate(); /* Remove devices from a DT overlay node */
  of_overlay_remove(); /* Remove the DT overlay node itself */

Some warnings are raised by __of_changeset_entry_destroy() which  was
called from of_overlay_remove():
  ERROR: memory leak, expected refcount 1 instead of 2 ...

The issue is that, during the device devlink removals triggered from the
of_platform_depopulate(), jobs are put in a workqueue.
These jobs drop the reference to the devices. When a device is no more
referenced (refcount == 0), it is released and the reference to its
of_node is dropped by a call to of_node_put().
These operations are fully correct except that, because of the
workqueue, they are done asynchronously with respect to function calls.

In the sequence provided, the jobs are run too late, after the call to
__of_changeset_entry_destroy() and so a missing of_node_put() call is
detected by __of_changeset_entry_destroy().

This series fixes this issue introducing device_link_wait_removal() in
order to wait for the end of jobs execution (patch 1) and using this
function to synchronize the overlay removal with the end of jobs
execution (patch 2).

Compared to the previous iteration:
  https://lore.kernel.org/linux-kernel/20240306085007.169771-1-herve.codina@bootlin.com/
this v5 series:
- Remove a 'Fixes' tag
- Update a comment
- Add 'Tested-by' and ''Reviewed-by' tags

This series handles cases reported by Luca [1] and Nuno [2].
  [1]: https://lore.kernel.org/all/20231220181627.341e8789@booty/
  [2]: https://lore.kernel.org/all/20240205-fix-device-links-overlays-v2-2-5344f8c79d57@analog.com/

Best regards,
Hervé

Changes v4 -> v5
  - Patch 1
    Remove the 'Fixes' tag
    Add 'Tested-by: Luca Ceresoli <luca.ceresoli@bootlin.com>'
    Add 'Reviewed-by: Nuno Sa <nuno.sa@analog.com>'

  - Patch 2
    Update comment as suggested
    Add 'Reviewed-by: Saravana Kannan <saravanak@google.com>'
    Add 'Tested-by: Luca Ceresoli <luca.ceresoli@bootlin.com>'
    Add 'Reviewed-by: Nuno Sa <nuno.sa@analog.com>'

Changes v3 -> v4
  - Patch 1
    Uses flush_workqueue() instead of drain_workqueue().

  - Patch 2
    Remove unlock/re-lock when calling device_link_wait_removal()
    Move device_link_wait_removal() call to of_changeset_destroy()
    Update commit log

Changes v2 -> v3
  - Patch 1
    No changes

  - Patch 2
    Add missing device.h

Changes v1 -> v2
  - Patch 1
    Rename the workqueue to 'device_link_wq'
    Add 'Fixes' tag and Cc stable

  - Patch 2
    Add device.h inclusion.
    Call device_link_wait_removal() later in the overlay removal
    sequence (i.e. in free_overlay_changeset() function).
    Drop of_mutex lock while calling device_link_wait_removal().
    Add	'Fixes'	tag and Cc stable

Herve Codina (2):
  driver core: Introduce device_link_wait_removal()
  of: dynamic: Synchronize of_changeset_destroy() with the devlink
    removals

 drivers/base/core.c    | 26 +++++++++++++++++++++++---
 drivers/of/dynamic.c   | 12 ++++++++++++
 include/linux/device.h |  1 +
 3 files changed, 36 insertions(+), 3 deletions(-)

-- 
2.43.0

[PATCH v5 1/2] driver core: Introduce device_link_wait_removal()

[Herve Codina]

The commit 80dd33cf72d1 ("drivers: base: Fix device link removal")
introduces a workqueue to release the consumer and supplier devices used
in the devlink.
In the job queued, devices are release and in turn, when all the
references to these devices are dropped, the release function of the
device itself is called.

Nothing is present to provide some synchronisation with this workqueue
in order to ensure that all ongoing releasing operations are done and
so, some other operations can be started safely.

For instance, in the following sequence:
  1) of_platform_depopulate()
  2) of_overlay_remove()

During the step 1, devices are released and related devlinks are removed
(jobs pushed in the workqueue).
During the step 2, OF nodes are destroyed but, without any
synchronisation with devlink removal jobs, of_overlay_remove() can raise
warnings related to missing of_node_put():
  ERROR: memory leak, expected refcount 1 instead of 2

Indeed, the missing of_node_put() call is going to be done, too late,
from the workqueue job execution.

Introduce device_link_wait_removal() to offer a way to synchronize
operations waiting for the end of devlink removals (i.e. end of
workqueue jobs).
Also, as a flushing operation is done on the workqueue, the workqueue
used is moved from a system-wide workqueue to a local one.

Cc: stable@vger.kernel.org
Signed-off-by: Herve Codina <herve.codina@bootlin.com>
Tested-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Reviewed-by: Nuno Sa <nuno.sa@analog.com>
---
 drivers/base/core.c    | 26 +++++++++++++++++++++++---
 include/linux/device.h |  1 +
 2 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/drivers/base/core.c b/drivers/base/core.c
index d5f4e4aac09b..48b28c59c592 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -44,6 +44,7 @@ static bool fw_devlink_is_permissive(void);
 static void __fw_devlink_link_to_consumers(struct device *dev);
 static bool fw_devlink_drv_reg_done;
 static bool fw_devlink_best_effort;
+static struct workqueue_struct *device_link_wq;
 
 /**
  * __fwnode_link_add - Create a link between two fwnode_handles.
@@ -532,12 +533,26 @@ static void devlink_dev_release(struct device *dev)
 	/*
 	 * It may take a while to complete this work because of the SRCU
 	 * synchronization in device_link_release_fn() and if the consumer or
-	 * supplier devices get deleted when it runs, so put it into the "long"
-	 * workqueue.
+	 * supplier devices get deleted when it runs, so put it into the
+	 * dedicated workqueue.
 	 */
-	queue_work(system_long_wq, &link->rm_work);
+	queue_work(device_link_wq, &link->rm_work);
 }
 
+/**
+ * device_link_wait_removal - Wait for ongoing devlink removal jobs to terminate
+ */
+void device_link_wait_removal(void)
+{
+	/*
+	 * devlink removal jobs are queued in the dedicated work queue.
+	 * To be sure that all removal jobs are terminated, ensure that any
+	 * scheduled work has run to completion.
+	 */
+	flush_workqueue(device_link_wq);
+}
+EXPORT_SYMBOL_GPL(device_link_wait_removal);
+
 static struct class devlink_class = {
 	.name = "devlink",
 	.dev_groups = devlink_groups,
@@ -4099,9 +4114,14 @@ int __init devices_init(void)
 	sysfs_dev_char_kobj = kobject_create_and_add("char", dev_kobj);
 	if (!sysfs_dev_char_kobj)
 		goto char_kobj_err;
+	device_link_wq = alloc_workqueue("device_link_wq", 0, 0);
+	if (!device_link_wq)
+		goto wq_err;
 
 	return 0;
 
+ wq_err:
+	kobject_put(sysfs_dev_char_kobj);
  char_kobj_err:
 	kobject_put(sysfs_dev_block_kobj);
  block_kobj_err:
diff --git a/include/linux/device.h b/include/linux/device.h
index 1795121dee9a..d7d8305a72e8 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -1249,6 +1249,7 @@ void device_link_del(struct device_link *link);
 void device_link_remove(void *consumer, struct device *supplier);
 void device_links_supplier_sync_state_pause(void);
 void device_links_supplier_sync_state_resume(void);
+void device_link_wait_removal(void);
 
 /* Create alias, so I can be autoloaded. */
 #define MODULE_ALIAS_CHARDEV(major,minor) \
-- 
2.43.0

[PATCH v5 2/2] of: dynamic: Synchronize of_changeset_destroy() with the devlink removals

[Herve Codina]

In the following sequence:
  1) of_platform_depopulate()
  2) of_overlay_remove()

During the step 1, devices are destroyed and devlinks are removed.
During the step 2, OF nodes are destroyed but
__of_changeset_entry_destroy() can raise warnings related to missing
of_node_put():
  ERROR: memory leak, expected refcount 1 instead of 2 ...

Indeed, during the devlink removals performed at step 1, the removal
itself releasing the device (and the attached of_node) is done by a job
queued in a workqueue and so, it is done asynchronously with respect to
function calls.
When the warning is present, of_node_put() will be called but wrongly
too late from the workqueue job.

In order to be sure that any ongoing devlink removals are done before
the of_node destruction, synchronize the of_changeset_destroy() with the
devlink removals.

Fixes: 80dd33cf72d1 ("drivers: base: Fix device link removal")
Cc: stable@vger.kernel.org
Signed-off-by: Herve Codina <herve.codina@bootlin.com>
Reviewed-by: Saravana Kannan <saravanak@google.com>
Tested-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Reviewed-by: Nuno Sa <nuno.sa@analog.com>
---
 drivers/of/dynamic.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/of/dynamic.c b/drivers/of/dynamic.c
index 3bf27052832f..4d57a4e34105 100644
--- a/drivers/of/dynamic.c
+++ b/drivers/of/dynamic.c
@@ -9,6 +9,7 @@
 
 #define pr_fmt(fmt)	"OF: " fmt
 
+#include <linux/device.h>
 #include <linux/of.h>
 #include <linux/spinlock.h>
 #include <linux/slab.h>
@@ -667,6 +668,17 @@ void of_changeset_destroy(struct of_changeset *ocs)
 {
 	struct of_changeset_entry *ce, *cen;
 
+	/*
+	 * When a device is deleted, the device links to/from it are also queued
+	 * for deletion. Until these device links are freed, the devices
+	 * themselves aren't freed. If the device being deleted is due to an
+	 * overlay change, this device might be holding a reference to a device
+	 * node that will be freed. So, wait until all already pending device
+	 * links are deleted before freeing a device node. This ensures we don't
+	 * free any device node that has a non-zero reference count.
+	 */
+	device_link_wait_removal();
+
 	list_for_each_entry_safe_reverse(ce, cen, &ocs->entries, node)
 		__of_changeset_entry_destroy(ce);
 }
-- 
2.43.0

Re: [PATCH v5 1/2] driver core: Introduce device_link_wait_removal()

[Nuno Sá]

Hi Herve,


On Thu, 2024-03-07 at 12:10 +0100, Herve Codina wrote:
> The commit 80dd33cf72d1 ("drivers: base: Fix device link removal")
> introduces a workqueue to release the consumer and supplier devices used
> in the devlink.
> In the job queued, devices are release and in turn, when all the
> references to these devices are dropped, the release function of the
> device itself is called.
> 
> Nothing is present to provide some synchronisation with this workqueue
> in order to ensure that all ongoing releasing operations are done and
> so, some other operations can be started safely.
> 
> For instance, in the following sequence:
>   1) of_platform_depopulate()
>   2) of_overlay_remove()
> 
> During the step 1, devices are released and related devlinks are removed
> (jobs pushed in the workqueue).
> During the step 2, OF nodes are destroyed but, without any
> synchronisation with devlink removal jobs, of_overlay_remove() can raise
> warnings related to missing of_node_put():
>   ERROR: memory leak, expected refcount 1 instead of 2
> 
> Indeed, the missing of_node_put() call is going to be done, too late,
> from the workqueue job execution.
> 
> Introduce device_link_wait_removal() to offer a way to synchronize
> operations waiting for the end of devlink removals (i.e. end of
> workqueue jobs).
> Also, as a flushing operation is done on the workqueue, the workqueue
> used is moved from a system-wide workqueue to a local one.
> 
> Cc: stable@vger.kernel.org
> Signed-off-by: Herve Codina <herve.codina@bootlin.com>
> Tested-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
> Reviewed-by: Nuno Sa <nuno.sa@analog.com>
> ---
>  drivers/base/core.c    | 26 +++++++++++++++++++++++---
>  include/linux/device.h |  1 +
>  2 files changed, 24 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/base/core.c b/drivers/base/core.c
> index d5f4e4aac09b..48b28c59c592 100644
> --- a/drivers/base/core.c
> +++ b/drivers/base/core.c
> @@ -44,6 +44,7 @@ static bool fw_devlink_is_permissive(void);
>  static void __fw_devlink_link_to_consumers(struct device *dev);
>  static bool fw_devlink_drv_reg_done;
>  static bool fw_devlink_best_effort;
> +static struct workqueue_struct *device_link_wq;
>  
>  /**
>   * __fwnode_link_add - Create a link between two fwnode_handles.
> @@ -532,12 +533,26 @@ static void devlink_dev_release(struct device *dev)
>  	/*
>  	 * It may take a while to complete this work because of the SRCU
>  	 * synchronization in device_link_release_fn() and if the consumer or
> -	 * supplier devices get deleted when it runs, so put it into the "long"
> -	 * workqueue.
> +	 * supplier devices get deleted when it runs, so put it into the
> +	 * dedicated workqueue.
>  	 */
> -	queue_work(system_long_wq, &link->rm_work);
> +	queue_work(device_link_wq, &link->rm_work);
>  }
>  
> +/**
> + * device_link_wait_removal - Wait for ongoing devlink removal jobs to terminate
> + */
> +void device_link_wait_removal(void)
> +{
> +	/*
> +	 * devlink removal jobs are queued in the dedicated work queue.
> +	 * To be sure that all removal jobs are terminated, ensure that any
> +	 * scheduled work has run to completion.
> +	 */
> +	flush_workqueue(device_link_wq);
> +}
> +EXPORT_SYMBOL_GPL(device_link_wait_removal);
> +
>  static struct class devlink_class = {
>  	.name = "devlink",
>  	.dev_groups = devlink_groups,
> @@ -4099,9 +4114,14 @@ int __init devices_init(void)
>  	sysfs_dev_char_kobj = kobject_create_and_add("char", dev_kobj);
>  	if (!sysfs_dev_char_kobj)
>  		goto char_kobj_err;
> +	device_link_wq = alloc_workqueue("device_link_wq", 0, 0);

My rb tag was with the assumption this is moved into devlink_class_init(). IIUC,
Saravana also agreed with that [1]. But it looks like he missed that we are
allocating the queue in devices_init() and not in devlink_class_init().

I'm also not sure if this is in line with what Rafael wanted for ccing stable. How do
we know the next patch depends on this one?

[1]: https://lore.kernel.org/lkml/CAGETcx_gNWOTsSZMaZu+XU1-5Z60WEcMhw08t4Sn_-YgkCCUmA@mail.gmail.com/

- Nuno Sá

Re: [PATCH v5 1/2] driver core: Introduce device_link_wait_removal()

[Nuno Sá]

On Thu, 2024-03-07 at 12:50 +0100, Nuno Sá wrote:
> Hi Herve,
> 
> 
> On Thu, 2024-03-07 at 12:10 +0100, Herve Codina wrote:
> > The commit 80dd33cf72d1 ("drivers: base: Fix device link removal")
> > introduces a workqueue to release the consumer and supplier devices used
> > in the devlink.
> > In the job queued, devices are release and in turn, when all the
> > references to these devices are dropped, the release function of the
> > device itself is called.
> > 
> > Nothing is present to provide some synchronisation with this workqueue
> > in order to ensure that all ongoing releasing operations are done and
> > so, some other operations can be started safely.
> > 
> > For instance, in the following sequence:
> >   1) of_platform_depopulate()
> >   2) of_overlay_remove()
> > 
> > During the step 1, devices are released and related devlinks are removed
> > (jobs pushed in the workqueue).
> > During the step 2, OF nodes are destroyed but, without any
> > synchronisation with devlink removal jobs, of_overlay_remove() can raise
> > warnings related to missing of_node_put():
> >   ERROR: memory leak, expected refcount 1 instead of 2
> > 
> > Indeed, the missing of_node_put() call is going to be done, too late,
> > from the workqueue job execution.
> > 
> > Introduce device_link_wait_removal() to offer a way to synchronize
> > operations waiting for the end of devlink removals (i.e. end of
> > workqueue jobs).
> > Also, as a flushing operation is done on the workqueue, the workqueue
> > used is moved from a system-wide workqueue to a local one.
> > 
> > Cc: stable@vger.kernel.org
> > Signed-off-by: Herve Codina <herve.codina@bootlin.com>
> > Tested-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
> > Reviewed-by: Nuno Sa <nuno.sa@analog.com>
> > ---
> >  drivers/base/core.c    | 26 +++++++++++++++++++++++---
> >  include/linux/device.h |  1 +
> >  2 files changed, 24 insertions(+), 3 deletions(-)
> > 
> > diff --git a/drivers/base/core.c b/drivers/base/core.c
> > index d5f4e4aac09b..48b28c59c592 100644
> > --- a/drivers/base/core.c
> > +++ b/drivers/base/core.c
> > @@ -44,6 +44,7 @@ static bool fw_devlink_is_permissive(void);
> >  static void __fw_devlink_link_to_consumers(struct device *dev);
> >  static bool fw_devlink_drv_reg_done;
> >  static bool fw_devlink_best_effort;
> > +static struct workqueue_struct *device_link_wq;
> >  
> >  /**
> >   * __fwnode_link_add - Create a link between two fwnode_handles.
> > @@ -532,12 +533,26 @@ static void devlink_dev_release(struct device *dev)
> >  	/*
> >  	 * It may take a while to complete this work because of the SRCU
> >  	 * synchronization in device_link_release_fn() and if the consumer or
> > -	 * supplier devices get deleted when it runs, so put it into the "long"
> > -	 * workqueue.
> > +	 * supplier devices get deleted when it runs, so put it into the
> > +	 * dedicated workqueue.
> >  	 */
> > -	queue_work(system_long_wq, &link->rm_work);
> > +	queue_work(device_link_wq, &link->rm_work);
> >  }
> >  
> > +/**
> > + * device_link_wait_removal - Wait for ongoing devlink removal jobs to terminate
> > + */
> > +void device_link_wait_removal(void)
> > +{
> > +	/*
> > +	 * devlink removal jobs are queued in the dedicated work queue.
> > +	 * To be sure that all removal jobs are terminated, ensure that any
> > +	 * scheduled work has run to completion.
> > +	 */
> > +	flush_workqueue(device_link_wq);
> > +}
> > +EXPORT_SYMBOL_GPL(device_link_wait_removal);
> > +
> >  static struct class devlink_class = {
> >  	.name = "devlink",
> >  	.dev_groups = devlink_groups,
> > @@ -4099,9 +4114,14 @@ int __init devices_init(void)
> >  	sysfs_dev_char_kobj = kobject_create_and_add("char", dev_kobj);
> >  	if (!sysfs_dev_char_kobj)
> >  		goto char_kobj_err;
> > +	device_link_wq = alloc_workqueue("device_link_wq", 0, 0);
> 
> My rb tag was with the assumption this is moved into devlink_class_init(). IIUC,
> Saravana also agreed with that [1]. But it looks like he missed that we are
> allocating the queue in devices_init() and not in devlink_class_init().
> 
> I'm also not sure if this is in line with what Rafael wanted for ccing stable. How
> do
> we know the next patch depends on this one?

Should have looked before:

https://www.kernel.org/doc/Documentation/process/stable-kernel-rules.rst

So, I think what you have is correct :)

- Nuno Sá

Re: [PATCH v5 1/2] driver core: Introduce device_link_wait_removal()

[Herve Codina]

Hi Nuno,

On Thu, 07 Mar 2024 12:50:52 +0100
Nuno Sá <noname.nuno@gmail.com> wrote:

> Hi Herve,
> 
> 
> On Thu, 2024-03-07 at 12:10 +0100, Herve Codina wrote:
> > The commit 80dd33cf72d1 ("drivers: base: Fix device link removal")
> > introduces a workqueue to release the consumer and supplier devices used
> > in the devlink.
> > In the job queued, devices are release and in turn, when all the
> > references to these devices are dropped, the release function of the
> > device itself is called.
> > 
> > Nothing is present to provide some synchronisation with this workqueue
> > in order to ensure that all ongoing releasing operations are done and
> > so, some other operations can be started safely.
> > 
> > For instance, in the following sequence:
> >   1) of_platform_depopulate()
> >   2) of_overlay_remove()
> > 
> > During the step 1, devices are released and related devlinks are removed
> > (jobs pushed in the workqueue).
> > During the step 2, OF nodes are destroyed but, without any
> > synchronisation with devlink removal jobs, of_overlay_remove() can raise
> > warnings related to missing of_node_put():
> >   ERROR: memory leak, expected refcount 1 instead of 2
> > 
> > Indeed, the missing of_node_put() call is going to be done, too late,
> > from the workqueue job execution.
> > 
> > Introduce device_link_wait_removal() to offer a way to synchronize
> > operations waiting for the end of devlink removals (i.e. end of
> > workqueue jobs).
> > Also, as a flushing operation is done on the workqueue, the workqueue
> > used is moved from a system-wide workqueue to a local one.
> > 
> > Cc: stable@vger.kernel.org
> > Signed-off-by: Herve Codina <herve.codina@bootlin.com>
> > Tested-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
> > Reviewed-by: Nuno Sa <nuno.sa@analog.com>
> > ---
> >  drivers/base/core.c    | 26 +++++++++++++++++++++++---
> >  include/linux/device.h |  1 +
> >  2 files changed, 24 insertions(+), 3 deletions(-)
> > 
> > diff --git a/drivers/base/core.c b/drivers/base/core.c
> > index d5f4e4aac09b..48b28c59c592 100644
> > --- a/drivers/base/core.c
> > +++ b/drivers/base/core.c
> > @@ -44,6 +44,7 @@ static bool fw_devlink_is_permissive(void);
> >  static void __fw_devlink_link_to_consumers(struct device *dev);
> >  static bool fw_devlink_drv_reg_done;
> >  static bool fw_devlink_best_effort;
> > +static struct workqueue_struct *device_link_wq;
> >  
> >  /**
> >   * __fwnode_link_add - Create a link between two fwnode_handles.
> > @@ -532,12 +533,26 @@ static void devlink_dev_release(struct device *dev)
> >  	/*
> >  	 * It may take a while to complete this work because of the SRCU
> >  	 * synchronization in device_link_release_fn() and if the consumer or
> > -	 * supplier devices get deleted when it runs, so put it into the "long"
> > -	 * workqueue.
> > +	 * supplier devices get deleted when it runs, so put it into the
> > +	 * dedicated workqueue.
> >  	 */
> > -	queue_work(system_long_wq, &link->rm_work);
> > +	queue_work(device_link_wq, &link->rm_work);
> >  }
> >  
> > +/**
> > + * device_link_wait_removal - Wait for ongoing devlink removal jobs to terminate
> > + */
> > +void device_link_wait_removal(void)
> > +{
> > +	/*
> > +	 * devlink removal jobs are queued in the dedicated work queue.
> > +	 * To be sure that all removal jobs are terminated, ensure that any
> > +	 * scheduled work has run to completion.
> > +	 */
> > +	flush_workqueue(device_link_wq);
> > +}
> > +EXPORT_SYMBOL_GPL(device_link_wait_removal);
> > +
> >  static struct class devlink_class = {
> >  	.name = "devlink",
> >  	.dev_groups = devlink_groups,
> > @@ -4099,9 +4114,14 @@ int __init devices_init(void)
> >  	sysfs_dev_char_kobj = kobject_create_and_add("char", dev_kobj);
> >  	if (!sysfs_dev_char_kobj)
> >  		goto char_kobj_err;
> > +	device_link_wq = alloc_workqueue("device_link_wq", 0, 0);  
> 
> My rb tag was with the assumption this is moved into devlink_class_init(). IIUC,
> Saravana also agreed with that [1]. But it looks like he missed that we are
> allocating the queue in devices_init() and not in devlink_class_init().
> 
> I'm also not sure if this is in line with what Rafael wanted for ccing stable. How do
> we know the next patch depends on this one?
> 
> [1]: https://lore.kernel.org/lkml/CAGETcx_gNWOTsSZMaZu+XU1-5Z60WEcMhw08t4Sn_-YgkCCUmA@mail.gmail.com/
> 

We discussed that point and I understood that you were ok to do that on your
side:
  https://lore.kernel.org/linux-kernel/f42ceee61ddb8b50c347589649d4131476ab5d81.camel@gmail.com/

Sorry if I misunderstood.

I am going to wait for other comments on this current series before re-sending
with our 'Reviewed-by' removed if needed. Let me know.

Best regards,
Hervé

Re: [PATCH v5 1/2] driver core: Introduce device_link_wait_removal()

[Nuno Sá]

On Thu, 2024-03-07 at 13:16 +0100, Herve Codina wrote:
> Hi Nuno,
> 
> On Thu, 07 Mar 2024 12:50:52 +0100
> Nuno Sá <noname.nuno@gmail.com> wrote:
> 
> > Hi Herve,
> > 
> > 
> > On Thu, 2024-03-07 at 12:10 +0100, Herve Codina wrote:
> > > The commit 80dd33cf72d1 ("drivers: base: Fix device link removal")
> > > introduces a workqueue to release the consumer and supplier devices used
> > > in the devlink.
> > > In the job queued, devices are release and in turn, when all the
> > > references to these devices are dropped, the release function of the
> > > device itself is called.
> > > 
> > > Nothing is present to provide some synchronisation with this workqueue
> > > in order to ensure that all ongoing releasing operations are done and
> > > so, some other operations can be started safely.
> > > 
> > > For instance, in the following sequence:
> > >   1) of_platform_depopulate()
> > >   2) of_overlay_remove()
> > > 
> > > During the step 1, devices are released and related devlinks are removed
> > > (jobs pushed in the workqueue).
> > > During the step 2, OF nodes are destroyed but, without any
> > > synchronisation with devlink removal jobs, of_overlay_remove() can raise
> > > warnings related to missing of_node_put():
> > >   ERROR: memory leak, expected refcount 1 instead of 2
> > > 
> > > Indeed, the missing of_node_put() call is going to be done, too late,
> > > from the workqueue job execution.
> > > 
> > > Introduce device_link_wait_removal() to offer a way to synchronize
> > > operations waiting for the end of devlink removals (i.e. end of
> > > workqueue jobs).
> > > Also, as a flushing operation is done on the workqueue, the workqueue
> > > used is moved from a system-wide workqueue to a local one.
> > > 
> > > Cc: stable@vger.kernel.org
> > > Signed-off-by: Herve Codina <herve.codina@bootlin.com>
> > > Tested-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
> > > Reviewed-by: Nuno Sa <nuno.sa@analog.com>
> > > ---
> > >  drivers/base/core.c    | 26 +++++++++++++++++++++++---
> > >  include/linux/device.h |  1 +
> > >  2 files changed, 24 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/drivers/base/core.c b/drivers/base/core.c
> > > index d5f4e4aac09b..48b28c59c592 100644
> > > --- a/drivers/base/core.c
> > > +++ b/drivers/base/core.c
> > > @@ -44,6 +44,7 @@ static bool fw_devlink_is_permissive(void);
> > >  static void __fw_devlink_link_to_consumers(struct device *dev);
> > >  static bool fw_devlink_drv_reg_done;
> > >  static bool fw_devlink_best_effort;
> > > +static struct workqueue_struct *device_link_wq;
> > >  
> > >  /**
> > >   * __fwnode_link_add - Create a link between two fwnode_handles.
> > > @@ -532,12 +533,26 @@ static void devlink_dev_release(struct device *dev)
> > >  	/*
> > >  	 * It may take a while to complete this work because of the SRCU
> > >  	 * synchronization in device_link_release_fn() and if the consumer or
> > > -	 * supplier devices get deleted when it runs, so put it into the
> > > "long"
> > > -	 * workqueue.
> > > +	 * supplier devices get deleted when it runs, so put it into the
> > > +	 * dedicated workqueue.
> > >  	 */
> > > -	queue_work(system_long_wq, &link->rm_work);
> > > +	queue_work(device_link_wq, &link->rm_work);
> > >  }
> > >  
> > > +/**
> > > + * device_link_wait_removal - Wait for ongoing devlink removal jobs to
> > > terminate
> > > + */
> > > +void device_link_wait_removal(void)
> > > +{
> > > +	/*
> > > +	 * devlink removal jobs are queued in the dedicated work queue.
> > > +	 * To be sure that all removal jobs are terminated, ensure that any
> > > +	 * scheduled work has run to completion.
> > > +	 */
> > > +	flush_workqueue(device_link_wq);
> > > +}
> > > +EXPORT_SYMBOL_GPL(device_link_wait_removal);
> > > +
> > >  static struct class devlink_class = {
> > >  	.name = "devlink",
> > >  	.dev_groups = devlink_groups,
> > > @@ -4099,9 +4114,14 @@ int __init devices_init(void)
> > >  	sysfs_dev_char_kobj = kobject_create_and_add("char", dev_kobj);
> > >  	if (!sysfs_dev_char_kobj)
> > >  		goto char_kobj_err;
> > > +	device_link_wq = alloc_workqueue("device_link_wq", 0, 0);  
> > 
> > My rb tag was with the assumption this is moved into devlink_class_init(). IIUC,
> > Saravana also agreed with that [1]. But it looks like he missed that we are
> > allocating the queue in devices_init() and not in devlink_class_init().
> > 
> > I'm also not sure if this is in line with what Rafael wanted for ccing stable.
> > How do
> > we know the next patch depends on this one?
> > 
> > [1]:
> > https://lore.kernel.org/lkml/CAGETcx_gNWOTsSZMaZu+XU1-5Z60WEcMhw08t4Sn_-YgkCCUmA@mail.gmail.com/
> > 
> 
> We discussed that point and I understood that you were ok to do that on your
> side:
>  
> https://lore.kernel.org/linux-kernel/f42ceee61ddb8b50c347589649d4131476ab5d81.camel@gmail.com/
> 
> Sorry if I misunderstood.

Oh, yeah, I can do that. But given Saravana reply I thought the expectation is to
have the queue already allocated in devlink_class_init().

> 
> I am going to wait for other comments on this current series before re-sending
> with our 'Reviewed-by' removed if needed. Let me know.
> 

Anyways, if your expectation was for me to do it later, fine. No need to remove the
tag. Sorry for the noise.

- Nuno Sá
> 

Re: [PATCH v5 0/2] Synchronize DT overlay removal with devlink removals

[Rob Herring]

On Thu, Mar 07, 2024 at 12:09:59PM +0100, Herve Codina wrote:
> Hi,
> 
> In the following sequence:
>   of_platform_depopulate(); /* Remove devices from a DT overlay node */
>   of_overlay_remove(); /* Remove the DT overlay node itself */
> 
> Some warnings are raised by __of_changeset_entry_destroy() which  was
> called from of_overlay_remove():
>   ERROR: memory leak, expected refcount 1 instead of 2 ...
> 
> The issue is that, during the device devlink removals triggered from the
> of_platform_depopulate(), jobs are put in a workqueue.
> These jobs drop the reference to the devices. When a device is no more
> referenced (refcount == 0), it is released and the reference to its
> of_node is dropped by a call to of_node_put().
> These operations are fully correct except that, because of the
> workqueue, they are done asynchronously with respect to function calls.
> 
> In the sequence provided, the jobs are run too late, after the call to
> __of_changeset_entry_destroy() and so a missing of_node_put() call is
> detected by __of_changeset_entry_destroy().
> 
> This series fixes this issue introducing device_link_wait_removal() in
> order to wait for the end of jobs execution (patch 1) and using this
> function to synchronize the overlay removal with the end of jobs
> execution (patch 2).
> 
> Compared to the previous iteration:
>   https://lore.kernel.org/linux-kernel/20240306085007.169771-1-herve.codina@bootlin.com/
> this v5 series:
> - Remove a 'Fixes' tag
> - Update a comment
> - Add 'Tested-by' and ''Reviewed-by' tags
> 
> This series handles cases reported by Luca [1] and Nuno [2].
>   [1]: https://lore.kernel.org/all/20231220181627.341e8789@booty/
>   [2]: https://lore.kernel.org/all/20240205-fix-device-links-overlays-v2-2-5344f8c79d57@analog.com/
> 
> Best regards,
> Hervé
> 
> Changes v4 -> v5
>   - Patch 1
>     Remove the 'Fixes' tag
>     Add 'Tested-by: Luca Ceresoli <luca.ceresoli@bootlin.com>'
>     Add 'Reviewed-by: Nuno Sa <nuno.sa@analog.com>'
> 
>   - Patch 2
>     Update comment as suggested
>     Add 'Reviewed-by: Saravana Kannan <saravanak@google.com>'
>     Add 'Tested-by: Luca Ceresoli <luca.ceresoli@bootlin.com>'
>     Add 'Reviewed-by: Nuno Sa <nuno.sa@analog.com>'
> 
> Changes v3 -> v4
>   - Patch 1
>     Uses flush_workqueue() instead of drain_workqueue().
> 
>   - Patch 2
>     Remove unlock/re-lock when calling device_link_wait_removal()
>     Move device_link_wait_removal() call to of_changeset_destroy()
>     Update commit log
> 
> Changes v2 -> v3
>   - Patch 1
>     No changes
> 
>   - Patch 2
>     Add missing device.h
> 
> Changes v1 -> v2
>   - Patch 1
>     Rename the workqueue to 'device_link_wq'
>     Add 'Fixes' tag and Cc stable
> 
>   - Patch 2
>     Add device.h inclusion.
>     Call device_link_wait_removal() later in the overlay removal
>     sequence (i.e. in free_overlay_changeset() function).
>     Drop of_mutex lock while calling device_link_wait_removal().
>     Add	'Fixes'	tag and Cc stable
> 
> Herve Codina (2):
>   driver core: Introduce device_link_wait_removal()
>   of: dynamic: Synchronize of_changeset_destroy() with the devlink
>     removals
> 
>  drivers/base/core.c    | 26 +++++++++++++++++++++++---
>  drivers/of/dynamic.c   | 12 ++++++++++++
>  include/linux/device.h |  1 +
>  3 files changed, 36 insertions(+), 3 deletions(-)

This looks good to me. I can take this given the user is DT. Looking for 
a R-by from Saravana and Ack from Greg. A R-by from Rafael would be 
great too.

Rob

Re: [PATCH v5 0/2] Synchronize DT overlay removal with devlink removals

[Saravana Kannan]

On Fri, Mar 8, 2024 at 12:05 PM Rob Herring <robh@kernel.org> wrote:
>
> On Thu, Mar 07, 2024 at 12:09:59PM +0100, Herve Codina wrote:
> > Hi,
> >
> > In the following sequence:
> >   of_platform_depopulate(); /* Remove devices from a DT overlay node */
> >   of_overlay_remove(); /* Remove the DT overlay node itself */
> >
> > Some warnings are raised by __of_changeset_entry_destroy() which  was
> > called from of_overlay_remove():
> >   ERROR: memory leak, expected refcount 1 instead of 2 ...
> >
> > The issue is that, during the device devlink removals triggered from the
> > of_platform_depopulate(), jobs are put in a workqueue.
> > These jobs drop the reference to the devices. When a device is no more
> > referenced (refcount == 0), it is released and the reference to its
> > of_node is dropped by a call to of_node_put().
> > These operations are fully correct except that, because of the
> > workqueue, they are done asynchronously with respect to function calls.
> >
> > In the sequence provided, the jobs are run too late, after the call to
> > __of_changeset_entry_destroy() and so a missing of_node_put() call is
> > detected by __of_changeset_entry_destroy().
> >
> > This series fixes this issue introducing device_link_wait_removal() in
> > order to wait for the end of jobs execution (patch 1) and using this
> > function to synchronize the overlay removal with the end of jobs
> > execution (patch 2).
> >
> > Compared to the previous iteration:
> >   https://lore.kernel.org/linux-kernel/20240306085007.169771-1-herve.codina@bootlin.com/
> > this v5 series:
> > - Remove a 'Fixes' tag
> > - Update a comment
> > - Add 'Tested-by' and ''Reviewed-by' tags
> >
> > This series handles cases reported by Luca [1] and Nuno [2].
> >   [1]: https://lore.kernel.org/all/20231220181627.341e8789@booty/
> >   [2]: https://lore.kernel.org/all/20240205-fix-device-links-overlays-v2-2-5344f8c79d57@analog.com/
> >
> > Best regards,
> > Hervé
> >
> > Changes v4 -> v5
> >   - Patch 1
> >     Remove the 'Fixes' tag
> >     Add 'Tested-by: Luca Ceresoli <luca.ceresoli@bootlin.com>'
> >     Add 'Reviewed-by: Nuno Sa <nuno.sa@analog.com>'
> >
> >   - Patch 2
> >     Update comment as suggested
> >     Add 'Reviewed-by: Saravana Kannan <saravanak@google.com>'
> >     Add 'Tested-by: Luca Ceresoli <luca.ceresoli@bootlin.com>'
> >     Add 'Reviewed-by: Nuno Sa <nuno.sa@analog.com>'
> >
> > Changes v3 -> v4
> >   - Patch 1
> >     Uses flush_workqueue() instead of drain_workqueue().
> >
> >   - Patch 2
> >     Remove unlock/re-lock when calling device_link_wait_removal()
> >     Move device_link_wait_removal() call to of_changeset_destroy()
> >     Update commit log
> >
> > Changes v2 -> v3
> >   - Patch 1
> >     No changes
> >
> >   - Patch 2
> >     Add missing device.h
> >
> > Changes v1 -> v2
> >   - Patch 1
> >     Rename the workqueue to 'device_link_wq'
> >     Add 'Fixes' tag and Cc stable
> >
> >   - Patch 2
> >     Add device.h inclusion.
> >     Call device_link_wait_removal() later in the overlay removal
> >     sequence (i.e. in free_overlay_changeset() function).
> >     Drop of_mutex lock while calling device_link_wait_removal().
> >     Add       'Fixes' tag and Cc stable
> >
> > Herve Codina (2):
> >   driver core: Introduce device_link_wait_removal()
> >   of: dynamic: Synchronize of_changeset_destroy() with the devlink
> >     removals
> >
> >  drivers/base/core.c    | 26 +++++++++++++++++++++++---
> >  drivers/of/dynamic.c   | 12 ++++++++++++
> >  include/linux/device.h |  1 +
> >  3 files changed, 36 insertions(+), 3 deletions(-)
>
> This looks good to me. I can take this given the user is DT. Looking for
> a R-by from Saravana and Ack from Greg. A R-by from Rafael would be
> great too.

Reviewed-by: Saravana Kannan <saravanak@google.com>

-Saravana

Re: [PATCH v5 0/2] Synchronize DT overlay removal with devlink removals

[Rob Herring]

On Fri, Mar 08, 2024 at 02:05:48PM -0600, Rob Herring wrote:
> On Thu, Mar 07, 2024 at 12:09:59PM +0100, Herve Codina wrote:
> > Hi,
> > 
> > In the following sequence:
> >   of_platform_depopulate(); /* Remove devices from a DT overlay node */
> >   of_overlay_remove(); /* Remove the DT overlay node itself */
> > 
> > Some warnings are raised by __of_changeset_entry_destroy() which  was
> > called from of_overlay_remove():
> >   ERROR: memory leak, expected refcount 1 instead of 2 ...
> > 
> > The issue is that, during the device devlink removals triggered from the
> > of_platform_depopulate(), jobs are put in a workqueue.
> > These jobs drop the reference to the devices. When a device is no more
> > referenced (refcount == 0), it is released and the reference to its
> > of_node is dropped by a call to of_node_put().
> > These operations are fully correct except that, because of the
> > workqueue, they are done asynchronously with respect to function calls.
> > 
> > In the sequence provided, the jobs are run too late, after the call to
> > __of_changeset_entry_destroy() and so a missing of_node_put() call is
> > detected by __of_changeset_entry_destroy().
> > 
> > This series fixes this issue introducing device_link_wait_removal() in
> > order to wait for the end of jobs execution (patch 1) and using this
> > function to synchronize the overlay removal with the end of jobs
> > execution (patch 2).
> > 
> > Compared to the previous iteration:
> >   https://lore.kernel.org/linux-kernel/20240306085007.169771-1-herve.codina@bootlin.com/
> > this v5 series:
> > - Remove a 'Fixes' tag
> > - Update a comment
> > - Add 'Tested-by' and ''Reviewed-by' tags
> > 
> > This series handles cases reported by Luca [1] and Nuno [2].
> >   [1]: https://lore.kernel.org/all/20231220181627.341e8789@booty/
> >   [2]: https://lore.kernel.org/all/20240205-fix-device-links-overlays-v2-2-5344f8c79d57@analog.com/
> > 
> > Best regards,
> > Hervé
> > 
> > Changes v4 -> v5
> >   - Patch 1
> >     Remove the 'Fixes' tag
> >     Add 'Tested-by: Luca Ceresoli <luca.ceresoli@bootlin.com>'
> >     Add 'Reviewed-by: Nuno Sa <nuno.sa@analog.com>'
> > 
> >   - Patch 2
> >     Update comment as suggested
> >     Add 'Reviewed-by: Saravana Kannan <saravanak@google.com>'
> >     Add 'Tested-by: Luca Ceresoli <luca.ceresoli@bootlin.com>'
> >     Add 'Reviewed-by: Nuno Sa <nuno.sa@analog.com>'
> > 
> > Changes v3 -> v4
> >   - Patch 1
> >     Uses flush_workqueue() instead of drain_workqueue().
> > 
> >   - Patch 2
> >     Remove unlock/re-lock when calling device_link_wait_removal()
> >     Move device_link_wait_removal() call to of_changeset_destroy()
> >     Update commit log
> > 
> > Changes v2 -> v3
> >   - Patch 1
> >     No changes
> > 
> >   - Patch 2
> >     Add missing device.h
> > 
> > Changes v1 -> v2
> >   - Patch 1
> >     Rename the workqueue to 'device_link_wq'
> >     Add 'Fixes' tag and Cc stable
> > 
> >   - Patch 2
> >     Add device.h inclusion.
> >     Call device_link_wait_removal() later in the overlay removal
> >     sequence (i.e. in free_overlay_changeset() function).
> >     Drop of_mutex lock while calling device_link_wait_removal().
> >     Add	'Fixes'	tag and Cc stable
> > 
> > Herve Codina (2):
> >   driver core: Introduce device_link_wait_removal()
> >   of: dynamic: Synchronize of_changeset_destroy() with the devlink
> >     removals
> > 
> >  drivers/base/core.c    | 26 +++++++++++++++++++++++---
> >  drivers/of/dynamic.c   | 12 ++++++++++++
> >  include/linux/device.h |  1 +
> >  3 files changed, 36 insertions(+), 3 deletions(-)
> 
> This looks good to me. I can take this given the user is DT. Looking for 
> a R-by from Saravana and Ack from Greg. A R-by from Rafael would be 
> great too.

You may want to resend this as Greg may have seen the discussion and 
moved on.

Rob

Re: [PATCH v5 0/2] Synchronize DT overlay removal with devlink removals

[Greg Kroah-Hartman]

On Fri, Mar 08, 2024 at 02:29:28PM -0800, Saravana Kannan wrote:
> On Fri, Mar 8, 2024 at 12:05 PM Rob Herring <robh@kernel.org> wrote:
> >
> > On Thu, Mar 07, 2024 at 12:09:59PM +0100, Herve Codina wrote:
> > > Hi,
> > >
> > > In the following sequence:
> > >   of_platform_depopulate(); /* Remove devices from a DT overlay node */
> > >   of_overlay_remove(); /* Remove the DT overlay node itself */
> > >
> > > Some warnings are raised by __of_changeset_entry_destroy() which  was
> > > called from of_overlay_remove():
> > >   ERROR: memory leak, expected refcount 1 instead of 2 ...
> > >
> > > The issue is that, during the device devlink removals triggered from the
> > > of_platform_depopulate(), jobs are put in a workqueue.
> > > These jobs drop the reference to the devices. When a device is no more
> > > referenced (refcount == 0), it is released and the reference to its
> > > of_node is dropped by a call to of_node_put().
> > > These operations are fully correct except that, because of the
> > > workqueue, they are done asynchronously with respect to function calls.
> > >
> > > In the sequence provided, the jobs are run too late, after the call to
> > > __of_changeset_entry_destroy() and so a missing of_node_put() call is
> > > detected by __of_changeset_entry_destroy().
> > >
> > > This series fixes this issue introducing device_link_wait_removal() in
> > > order to wait for the end of jobs execution (patch 1) and using this
> > > function to synchronize the overlay removal with the end of jobs
> > > execution (patch 2).
> > >
> > > Compared to the previous iteration:
> > >   https://lore.kernel.org/linux-kernel/20240306085007.169771-1-herve.codina@bootlin.com/
> > > this v5 series:
> > > - Remove a 'Fixes' tag
> > > - Update a comment
> > > - Add 'Tested-by' and ''Reviewed-by' tags
> > >
> > > This series handles cases reported by Luca [1] and Nuno [2].
> > >   [1]: https://lore.kernel.org/all/20231220181627.341e8789@booty/
> > >   [2]: https://lore.kernel.org/all/20240205-fix-device-links-overlays-v2-2-5344f8c79d57@analog.com/
> > >
> > > Best regards,
> > > Hervé
> > >
> > > Changes v4 -> v5
> > >   - Patch 1
> > >     Remove the 'Fixes' tag
> > >     Add 'Tested-by: Luca Ceresoli <luca.ceresoli@bootlin.com>'
> > >     Add 'Reviewed-by: Nuno Sa <nuno.sa@analog.com>'
> > >
> > >   - Patch 2
> > >     Update comment as suggested
> > >     Add 'Reviewed-by: Saravana Kannan <saravanak@google.com>'
> > >     Add 'Tested-by: Luca Ceresoli <luca.ceresoli@bootlin.com>'
> > >     Add 'Reviewed-by: Nuno Sa <nuno.sa@analog.com>'
> > >
> > > Changes v3 -> v4
> > >   - Patch 1
> > >     Uses flush_workqueue() instead of drain_workqueue().
> > >
> > >   - Patch 2
> > >     Remove unlock/re-lock when calling device_link_wait_removal()
> > >     Move device_link_wait_removal() call to of_changeset_destroy()
> > >     Update commit log
> > >
> > > Changes v2 -> v3
> > >   - Patch 1
> > >     No changes
> > >
> > >   - Patch 2
> > >     Add missing device.h
> > >
> > > Changes v1 -> v2
> > >   - Patch 1
> > >     Rename the workqueue to 'device_link_wq'
> > >     Add 'Fixes' tag and Cc stable
> > >
> > >   - Patch 2
> > >     Add device.h inclusion.
> > >     Call device_link_wait_removal() later in the overlay removal
> > >     sequence (i.e. in free_overlay_changeset() function).
> > >     Drop of_mutex lock while calling device_link_wait_removal().
> > >     Add       'Fixes' tag and Cc stable
> > >
> > > Herve Codina (2):
> > >   driver core: Introduce device_link_wait_removal()
> > >   of: dynamic: Synchronize of_changeset_destroy() with the devlink
> > >     removals
> > >
> > >  drivers/base/core.c    | 26 +++++++++++++++++++++++---
> > >  drivers/of/dynamic.c   | 12 ++++++++++++
> > >  include/linux/device.h |  1 +
> > >  3 files changed, 36 insertions(+), 3 deletions(-)
> >
> > This looks good to me. I can take this given the user is DT. Looking for
> > a R-by from Saravana and Ack from Greg. A R-by from Rafael would be
> > great too.
> 
> Reviewed-by: Saravana Kannan <saravanak@google.com>

Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>