[PATCH 0/2] fix DT overlays when device links are released

[PATCH 0/2] fix DT overlays when device links are released

[Nuno Sa via B4 Relay]

Link to RFC:
 * https://lore.kernel.org/lkml/20240123-fix-device-links-overlays-v1-1-9e4f6acaab6c@analog.com/

Changes since RFC:
 * Use a dedicated workqueue to remove devlinks;
 * Flush the devlink workqueue before checking the of_node refcount
   value.

The following series is the result of the discussion I had with Rafael.
To sum up the fundamental issue, device links drop their refcounts
asynchronously and that means that the of_node refcount associated with
the device will also be dropped asynchronously. Now, in
__of_changeset_entry_destroy(), the assumption is that the node refcount
must be 1 and that cannot be guaranteed given the above.

I'm pasting again the link of the first time I exposed the issue where
one can see the resulps (big splat) of failing DT assumption:

https://lore.kernel.org/linux-devicetree/20230511151047.1779841-1-nuno.sa@analog.com/

---
Nuno Sa (2):
      driver: core: add dedicated workqueue for devlink removal
      of: dynamic: flush devlinks workqueue before destroying the changeset

 drivers/base/core.c    | 33 +++++++++++++++++++++++++++++----
 drivers/of/dynamic.c   |  8 ++++++++
 include/linux/fwnode.h |  1 +
 3 files changed, 38 insertions(+), 4 deletions(-)
---
base-commit: 6613476e225e090cc9aad49be7fa504e290dd33d
change-id: 20240123-fix-device-links-overlays-5422e033a09b
--

Thanks!
- Nuno Sá

[PATCH 1/2] driver: core: add dedicated workqueue for devlink removal

[Nuno Sa via B4 Relay]

From: Nuno Sa <nuno.sa@analog.com>

Let's use a dedicated queue for devlinks since releasing a link happens
asynchronously but some code paths, like DT overlays, have some
expectations regarding the of_node when being removed (the refcount must
be 1). Given how devlinks are released that cannot be assured. Hence, add a
dedicated queue so that it's easy to sync against devlinks removal.

While at it, make sure to explicitly include <linux/workqueue.h>.

Signed-off-by: Nuno Sa <nuno.sa@analog.com>
---
 drivers/base/core.c    | 33 +++++++++++++++++++++++++++++----
 include/linux/fwnode.h |  1 +
 2 files changed, 30 insertions(+), 4 deletions(-)

diff --git a/drivers/base/core.c b/drivers/base/core.c
index 14d46af40f9a..06e7766b5227 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -31,6 +31,7 @@
 #include <linux/swiotlb.h>
 #include <linux/sysfs.h>
 #include <linux/dma-map-ops.h> /* for dma_default_coherent */
+#include <linux/workqueue.h>
 
 #include "base.h"
 #include "physical_location.h"
@@ -44,6 +45,7 @@ static bool fw_devlink_is_permissive(void);
 static void __fw_devlink_link_to_consumers(struct device *dev);
 static bool fw_devlink_drv_reg_done;
 static bool fw_devlink_best_effort;
+static struct workqueue_struct *devlink_release_queue __ro_after_init;
 
 /**
  * __fwnode_link_add - Create a link between two fwnode_handles.
@@ -235,6 +237,11 @@ static void __fw_devlink_pickup_dangling_consumers(struct fwnode_handle *fwnode,
 		__fw_devlink_pickup_dangling_consumers(child, new_sup);
 }
 
+void fwnode_links_flush_queue(void)
+{
+	flush_workqueue(devlink_release_queue);
+}
+
 static DEFINE_MUTEX(device_links_lock);
 DEFINE_STATIC_SRCU(device_links_srcu);
 
@@ -531,9 +538,10 @@ static void devlink_dev_release(struct device *dev)
 	 * It may take a while to complete this work because of the SRCU
 	 * synchronization in device_link_release_fn() and if the consumer or
 	 * supplier devices get deleted when it runs, so put it into the "long"
-	 * workqueue.
+	 * devlink workqueue.
+	 *
 	 */
-	queue_work(system_long_wq, &link->rm_work);
+	queue_work(devlink_release_queue, &link->rm_work);
 }
 
 static struct class devlink_class = {
@@ -636,10 +644,27 @@ static int __init devlink_class_init(void)
 		return ret;
 
 	ret = class_interface_register(&devlink_class_intf);
-	if (ret)
+	if (ret) {
+		class_unregister(&devlink_class);
+		return ret;
+	}
+
+	/*
+	 * Using a dedicated queue for devlinks since releasing a link happens
+	 * asynchronously but some code paths, like DT overlays, have some
+	 * expectations regarding the of_node when being removed (the refcount
+	 * must be 1). Given how devlinks are released that cannot be assured.
+	 * Hence, add a dedicated queue so that it's easy to sync against
+	 * devlinks removal.
+	 */
+	devlink_release_queue = alloc_workqueue("devlink_release", 0, 0);
+	if (!devlink_release_queue) {
+		class_interface_unregister(&devlink_class_intf);
 		class_unregister(&devlink_class);
+		return -ENODEV;
+	}
 
-	return ret;
+	return 0;
 }
 postcore_initcall(devlink_class_init);
 
diff --git a/include/linux/fwnode.h b/include/linux/fwnode.h
index 2a72f55d26eb..017b170e9903 100644
--- a/include/linux/fwnode.h
+++ b/include/linux/fwnode.h
@@ -213,5 +213,6 @@ extern bool fw_devlink_is_strict(void);
 int fwnode_link_add(struct fwnode_handle *con, struct fwnode_handle *sup);
 void fwnode_links_purge(struct fwnode_handle *fwnode);
 void fw_devlink_purge_absent_suppliers(struct fwnode_handle *fwnode);
+void fwnode_links_flush_queue(void);
 
 #endif

-- 
2.43.0

Re: [PATCH 1/2] driver: core: add dedicated workqueue for devlink removal

[Rafael J. Wysocki]

On Fri, Feb 2, 2024 at 1:18 PM Nuno Sa via B4 Relay
<devnull+nuno.sa.analog.com@kernel.org> wrote:
>
> From: Nuno Sa <nuno.sa@analog.com>
>
> Let's use a dedicated queue for devlinks since releasing a link happens
> asynchronously but some code paths, like DT overlays, have some
> expectations regarding the of_node when being removed (the refcount must
> be 1). Given how devlinks are released that cannot be assured. Hence, add a
> dedicated queue so that it's easy to sync against devlinks removal.

Thanks for following my suggestion!

> While at it, make sure to explicitly include <linux/workqueue.h>.
>
> Signed-off-by: Nuno Sa <nuno.sa@analog.com>
> ---
>  drivers/base/core.c    | 33 +++++++++++++++++++++++++++++----
>  include/linux/fwnode.h |  1 +
>  2 files changed, 30 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/base/core.c b/drivers/base/core.c
> index 14d46af40f9a..06e7766b5227 100644
> --- a/drivers/base/core.c
> +++ b/drivers/base/core.c
> @@ -31,6 +31,7 @@
>  #include <linux/swiotlb.h>
>  #include <linux/sysfs.h>
>  #include <linux/dma-map-ops.h> /* for dma_default_coherent */
> +#include <linux/workqueue.h>
>
>  #include "base.h"
>  #include "physical_location.h"
> @@ -44,6 +45,7 @@ static bool fw_devlink_is_permissive(void);
>  static void __fw_devlink_link_to_consumers(struct device *dev);
>  static bool fw_devlink_drv_reg_done;
>  static bool fw_devlink_best_effort;
> +static struct workqueue_struct *devlink_release_queue __ro_after_init;
>
>  /**
>   * __fwnode_link_add - Create a link between two fwnode_handles.
> @@ -235,6 +237,11 @@ static void __fw_devlink_pickup_dangling_consumers(struct fwnode_handle *fwnode,
>                 __fw_devlink_pickup_dangling_consumers(child, new_sup);
>  }
>
> +void fwnode_links_flush_queue(void)
> +{
> +       flush_workqueue(devlink_release_queue);
> +}
> +
>  static DEFINE_MUTEX(device_links_lock);
>  DEFINE_STATIC_SRCU(device_links_srcu);
>
> @@ -531,9 +538,10 @@ static void devlink_dev_release(struct device *dev)
>          * It may take a while to complete this work because of the SRCU
>          * synchronization in device_link_release_fn() and if the consumer or
>          * supplier devices get deleted when it runs, so put it into the "long"
> -        * workqueue.
> +        * devlink workqueue.
> +        *
>          */
> -       queue_work(system_long_wq, &link->rm_work);
> +       queue_work(devlink_release_queue, &link->rm_work);
>  }
>
>  static struct class devlink_class = {
> @@ -636,10 +644,27 @@ static int __init devlink_class_init(void)
>                 return ret;
>
>         ret = class_interface_register(&devlink_class_intf);
> -       if (ret)
> +       if (ret) {
> +               class_unregister(&devlink_class);
> +               return ret;
> +       }
> +
> +       /*
> +        * Using a dedicated queue for devlinks since releasing a link happens
> +        * asynchronously but some code paths, like DT overlays, have some
> +        * expectations regarding the of_node when being removed (the refcount
> +        * must be 1). Given how devlinks are released that cannot be assured.
> +        * Hence, add a dedicated queue so that it's easy to sync against
> +        * devlinks removal.
> +        */
> +       devlink_release_queue = alloc_workqueue("devlink_release", 0, 0);
> +       if (!devlink_release_queue) {
> +               class_interface_unregister(&devlink_class_intf);
>                 class_unregister(&devlink_class);

This is a bit drastic.

I think that device links can still work if devlink_release_queue is
NULL, just devlink_dev_release() needs to check it and release
synchronously if it is NULL.

Apart from this LGTM.

> +               return -ENODEV;
> +       }
>
> -       return ret;
> +       return 0;
>  }
>  postcore_initcall(devlink_class_init);
>
> diff --git a/include/linux/fwnode.h b/include/linux/fwnode.h
> index 2a72f55d26eb..017b170e9903 100644
> --- a/include/linux/fwnode.h
> +++ b/include/linux/fwnode.h
> @@ -213,5 +213,6 @@ extern bool fw_devlink_is_strict(void);
>  int fwnode_link_add(struct fwnode_handle *con, struct fwnode_handle *sup);
>  void fwnode_links_purge(struct fwnode_handle *fwnode);
>  void fw_devlink_purge_absent_suppliers(struct fwnode_handle *fwnode);
> +void fwnode_links_flush_queue(void);
>
>  #endif
>
> --

Re: [PATCH 1/2] driver: core: add dedicated workqueue for devlink removal

[Nuno Sá]

On Fri, 2024-02-02 at 16:59 +0100, Rafael J. Wysocki wrote:
> On Fri, Feb 2, 2024 at 1:18 PM Nuno Sa via B4 Relay
> <devnull+nuno.sa.analog.com@kernel.org> wrote:
> > 
> > From: Nuno Sa <nuno.sa@analog.com>
> > 
> > Let's use a dedicated queue for devlinks since releasing a link happens
> > asynchronously but some code paths, like DT overlays, have some
> > expectations regarding the of_node when being removed (the refcount must
> > be 1). Given how devlinks are released that cannot be assured. Hence, add a
> > dedicated queue so that it's easy to sync against devlinks removal.
> 
> Thanks for following my suggestion!
> 
> > While at it, make sure to explicitly include <linux/workqueue.h>.
> > 
> > Signed-off-by: Nuno Sa <nuno.sa@analog.com>
> > ---
> >  drivers/base/core.c    | 33 +++++++++++++++++++++++++++++----
> >  include/linux/fwnode.h |  1 +
> >  2 files changed, 30 insertions(+), 4 deletions(-)
> > 
> > diff --git a/drivers/base/core.c b/drivers/base/core.c
> > index 14d46af40f9a..06e7766b5227 100644
> > --- a/drivers/base/core.c
> > +++ b/drivers/base/core.c
> > @@ -31,6 +31,7 @@
> >  #include <linux/swiotlb.h>
> >  #include <linux/sysfs.h>
> >  #include <linux/dma-map-ops.h> /* for dma_default_coherent */
> > +#include <linux/workqueue.h>
> > 
> >  #include "base.h"
> >  #include "physical_location.h"
> > @@ -44,6 +45,7 @@ static bool fw_devlink_is_permissive(void);
> >  static void __fw_devlink_link_to_consumers(struct device *dev);
> >  static bool fw_devlink_drv_reg_done;
> >  static bool fw_devlink_best_effort;
> > +static struct workqueue_struct *devlink_release_queue __ro_after_init;
> > 
> >  /**
> >   * __fwnode_link_add - Create a link between two fwnode_handles.
> > @@ -235,6 +237,11 @@ static void __fw_devlink_pickup_dangling_consumers(struct
> > fwnode_handle *fwnode,
> >                 __fw_devlink_pickup_dangling_consumers(child, new_sup);
> >  }
> > 
> > +void fwnode_links_flush_queue(void)
> > +{
> > +       flush_workqueue(devlink_release_queue);
> > +}
> > +
> >  static DEFINE_MUTEX(device_links_lock);
> >  DEFINE_STATIC_SRCU(device_links_srcu);
> > 
> > @@ -531,9 +538,10 @@ static void devlink_dev_release(struct device *dev)
> >          * It may take a while to complete this work because of the SRCU
> >          * synchronization in device_link_release_fn() and if the consumer or
> >          * supplier devices get deleted when it runs, so put it into the "long"
> > -        * workqueue.
> > +        * devlink workqueue.
> > +        *
> >          */
> > -       queue_work(system_long_wq, &link->rm_work);
> > +       queue_work(devlink_release_queue, &link->rm_work);
> >  }
> > 
> >  static struct class devlink_class = {
> > @@ -636,10 +644,27 @@ static int __init devlink_class_init(void)
> >                 return ret;
> > 
> >         ret = class_interface_register(&devlink_class_intf);
> > -       if (ret)
> > +       if (ret) {
> > +               class_unregister(&devlink_class);
> > +               return ret;
> > +       }
> > +
> > +       /*
> > +        * Using a dedicated queue for devlinks since releasing a link happens
> > +        * asynchronously but some code paths, like DT overlays, have some
> > +        * expectations regarding the of_node when being removed (the refcount
> > +        * must be 1). Given how devlinks are released that cannot be assured.
> > +        * Hence, add a dedicated queue so that it's easy to sync against
> > +        * devlinks removal.
> > +        */
> > +       devlink_release_queue = alloc_workqueue("devlink_release", 0, 0);
> > +       if (!devlink_release_queue) {
> > +               class_interface_unregister(&devlink_class_intf);
> >                 class_unregister(&devlink_class);
> 
> This is a bit drastic.
> 
> I think that device links can still work if devlink_release_queue is
> NULL, just devlink_dev_release() needs to check it and release
> synchronously if it is NULL.
> 

Agreed, I'll do that way. It will always synchronously remove the links (which is
different than before) but I guess that failing in allocating the queue is rather
unlikely.

- Nuno Sá

Re: [PATCH 1/2] driver: core: add dedicated workqueue for devlink removal

[Rafael J. Wysocki]

On Mon, Feb 5, 2024 at 9:29 AM Nuno Sá <noname.nuno@gmail.com> wrote:
>
> On Fri, 2024-02-02 at 16:59 +0100, Rafael J. Wysocki wrote:
> > On Fri, Feb 2, 2024 at 1:18 PM Nuno Sa via B4 Relay
> > <devnull+nuno.sa.analog.com@kernel.org> wrote:
> > >
> > > From: Nuno Sa <nuno.sa@analog.com>
> > >
> > > Let's use a dedicated queue for devlinks since releasing a link happens
> > > asynchronously but some code paths, like DT overlays, have some
> > > expectations regarding the of_node when being removed (the refcount must
> > > be 1). Given how devlinks are released that cannot be assured. Hence, add a
> > > dedicated queue so that it's easy to sync against devlinks removal.
> >
> > Thanks for following my suggestion!
> >
> > > While at it, make sure to explicitly include <linux/workqueue.h>.
> > >
> > > Signed-off-by: Nuno Sa <nuno.sa@analog.com>
> > > ---
> > >  drivers/base/core.c    | 33 +++++++++++++++++++++++++++++----
> > >  include/linux/fwnode.h |  1 +
> > >  2 files changed, 30 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/drivers/base/core.c b/drivers/base/core.c
> > > index 14d46af40f9a..06e7766b5227 100644
> > > --- a/drivers/base/core.c
> > > +++ b/drivers/base/core.c
> > > @@ -31,6 +31,7 @@
> > >  #include <linux/swiotlb.h>
> > >  #include <linux/sysfs.h>
> > >  #include <linux/dma-map-ops.h> /* for dma_default_coherent */
> > > +#include <linux/workqueue.h>
> > >
> > >  #include "base.h"
> > >  #include "physical_location.h"
> > > @@ -44,6 +45,7 @@ static bool fw_devlink_is_permissive(void);
> > >  static void __fw_devlink_link_to_consumers(struct device *dev);
> > >  static bool fw_devlink_drv_reg_done;
> > >  static bool fw_devlink_best_effort;
> > > +static struct workqueue_struct *devlink_release_queue __ro_after_init;
> > >
> > >  /**
> > >   * __fwnode_link_add - Create a link between two fwnode_handles.
> > > @@ -235,6 +237,11 @@ static void __fw_devlink_pickup_dangling_consumers(struct
> > > fwnode_handle *fwnode,
> > >                 __fw_devlink_pickup_dangling_consumers(child, new_sup);
> > >  }
> > >
> > > +void fwnode_links_flush_queue(void)
> > > +{
> > > +       flush_workqueue(devlink_release_queue);
> > > +}
> > > +
> > >  static DEFINE_MUTEX(device_links_lock);
> > >  DEFINE_STATIC_SRCU(device_links_srcu);
> > >
> > > @@ -531,9 +538,10 @@ static void devlink_dev_release(struct device *dev)
> > >          * It may take a while to complete this work because of the SRCU
> > >          * synchronization in device_link_release_fn() and if the consumer or
> > >          * supplier devices get deleted when it runs, so put it into the "long"
> > > -        * workqueue.
> > > +        * devlink workqueue.
> > > +        *
> > >          */
> > > -       queue_work(system_long_wq, &link->rm_work);
> > > +       queue_work(devlink_release_queue, &link->rm_work);
> > >  }
> > >
> > >  static struct class devlink_class = {
> > > @@ -636,10 +644,27 @@ static int __init devlink_class_init(void)
> > >                 return ret;
> > >
> > >         ret = class_interface_register(&devlink_class_intf);
> > > -       if (ret)
> > > +       if (ret) {
> > > +               class_unregister(&devlink_class);
> > > +               return ret;
> > > +       }
> > > +
> > > +       /*
> > > +        * Using a dedicated queue for devlinks since releasing a link happens
> > > +        * asynchronously but some code paths, like DT overlays, have some
> > > +        * expectations regarding the of_node when being removed (the refcount
> > > +        * must be 1). Given how devlinks are released that cannot be assured.
> > > +        * Hence, add a dedicated queue so that it's easy to sync against
> > > +        * devlinks removal.
> > > +        */
> > > +       devlink_release_queue = alloc_workqueue("devlink_release", 0, 0);
> > > +       if (!devlink_release_queue) {
> > > +               class_interface_unregister(&devlink_class_intf);
> > >                 class_unregister(&devlink_class);
> >
> > This is a bit drastic.
> >
> > I think that device links can still work if devlink_release_queue is
> > NULL, just devlink_dev_release() needs to check it and release
> > synchronously if it is NULL.
> >
>
> Agreed, I'll do that way. It will always synchronously remove the links (which is
> different than before) but I guess that failing in allocating the queue is rather
> unlikely.

Right.

[PATCH 2/2] of: dynamic: flush devlinks workqueue before destroying the changeset

[Nuno Sa via B4 Relay]

From: Nuno Sa <nuno.sa@analog.com>

Device links will drop their supplier + consumer refcounts
asynchronously. That means that the refcount of the of_node attached to
these devices will also be dropped asynchronously and so we cannot
guarantee the DT overlay assumption that the of_node refcount must be 1 in
__of_changeset_entry_destroy().

Given the above, call the new fwnode_links_flush_queue() helper to flush
the devlink workqueue so we can be sure that all links are dropped before
doing the proper checks.

Signed-off-by: Nuno Sa <nuno.sa@analog.com>
---
 drivers/of/dynamic.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/of/dynamic.c b/drivers/of/dynamic.c
index 3bf27052832f..b7153c72c9c9 100644
--- a/drivers/of/dynamic.c
+++ b/drivers/of/dynamic.c
@@ -14,6 +14,7 @@
 #include <linux/slab.h>
 #include <linux/string.h>
 #include <linux/proc_fs.h>
+#include <linux/fwnode.h>
 
 #include "of_private.h"
 
@@ -518,6 +519,13 @@ EXPORT_SYMBOL(of_changeset_create_node);
 
 static void __of_changeset_entry_destroy(struct of_changeset_entry *ce)
 {
+	/*
+	 * device links drop their device references (and hence their of_node
+	 * references) asynchronously on a dedicated workqueue. Hence we need
+	 * to flush it to make sure everything is done before doing the below
+	 * checks.
+	 */
+	fwnode_links_flush_queue();
 	if (ce->action == OF_RECONFIG_ATTACH_NODE &&
 	    of_node_check_flag(ce->np, OF_OVERLAY)) {
 		if (kref_read(&ce->np->kobj.kref) > 1) {

-- 
2.43.0

Re: [PATCH 1/2] driver: core: add dedicated workqueue for devlink removal

[Andy Shevchenko]

On Fri, Feb 02, 2024 at 01:18:16PM +0100, Nuno Sa via B4 Relay wrote:
> From: Nuno Sa <nuno.sa@analog.com>
> 
> Let's use a dedicated queue for devlinks since releasing a link happens
> asynchronously but some code paths, like DT overlays, have some
> expectations regarding the of_node when being removed (the refcount must
> be 1). Given how devlinks are released that cannot be assured. Hence, add a
> dedicated queue so that it's easy to sync against devlinks removal.
> 
> While at it, make sure to explicitly include <linux/workqueue.h>.

...

> --- a/include/linux/fwnode.h
> +++ b/include/linux/fwnode.h
> @@ -213,5 +213,6 @@ extern bool fw_devlink_is_strict(void);
>  int fwnode_link_add(struct fwnode_handle *con, struct fwnode_handle *sup);
>  void fwnode_links_purge(struct fwnode_handle *fwnode);
>  void fw_devlink_purge_absent_suppliers(struct fwnode_handle *fwnode);
> +void fwnode_links_flush_queue(void);

Can it be grouped like

  void fwnode_links_flush_queue(void);
  void fwnode_links_purge(struct fwnode_handle *fwnode);
  void fw_devlink_purge_absent_suppliers(struct fwnode_handle *fwnode);

?

(devlinks namespaces seems a bit messy to me, but I'm not familiar with that
 code at all)

-- 
With Best Regards,
Andy Shevchenko