[PATCH net v2] ice: Unregister netdev and devlink_port only once

[PATCH net v2] ice: Unregister netdev and devlink_port only once

[Petr Oros]

Since commit 6624e780a577fc ("ice: split ice_vsi_setup into smaller
functions") ice_vsi_release does things twice. There is unregister
netdev which is unregistered in ice_deinit_eth also.

It also unregisters the devlink_port twice which is also unregistered
in ice_deinit_eth(). This double deregistration is hidden because
devl_port_unregister ignores the return value of xa_erase.

[   68.642167] Call Trace:
[   68.650385]  ice_devlink_destroy_pf_port+0xe/0x20 [ice]
[   68.655656]  ice_vsi_release+0x445/0x690 [ice]
[   68.660147]  ice_deinit+0x99/0x280 [ice]
[   68.664117]  ice_remove+0x1b6/0x5c0 [ice]

[  171.103841] Call Trace:
[  171.109607]  ice_devlink_destroy_pf_port+0xf/0x20 [ice]
[  171.114841]  ice_remove+0x158/0x270 [ice]
[  171.118854]  pci_device_remove+0x3b/0xc0
[  171.122779]  device_release_driver_internal+0xc7/0x170
[  171.127912]  driver_detach+0x54/0x8c
[  171.131491]  bus_remove_driver+0x77/0xd1
[  171.135406]  pci_unregister_driver+0x2d/0xb0
[  171.139670]  ice_module_exit+0xc/0x55f [ice]

Fixes: 6624e780a577 ("ice: split ice_vsi_setup into smaller functions")
Signed-off-by: Petr Oros <poros@redhat.com>
---
v2: reword subject

v1: https://lore.kernel.org/netdev/20230619084948.360128-1-poros@redhat.com/
---
 drivers/net/ethernet/intel/ice/ice_lib.c | 27 ------------------------
 1 file changed, 27 deletions(-)

diff --git a/drivers/net/ethernet/intel/ice/ice_lib.c b/drivers/net/ethernet/intel/ice/ice_lib.c
index 11ae0e41f518a1..284a1f0bfdb545 100644
--- a/drivers/net/ethernet/intel/ice/ice_lib.c
+++ b/drivers/net/ethernet/intel/ice/ice_lib.c
@@ -3272,39 +3272,12 @@ int ice_vsi_release(struct ice_vsi *vsi)
 		return -ENODEV;
 	pf = vsi->back;
 
-	/* do not unregister while driver is in the reset recovery pending
-	 * state. Since reset/rebuild happens through PF service task workqueue,
-	 * it's not a good idea to unregister netdev that is associated to the
-	 * PF that is running the work queue items currently. This is done to
-	 * avoid check_flush_dependency() warning on this wq
-	 */
-	if (vsi->netdev && !ice_is_reset_in_progress(pf->state) &&
-	    (test_bit(ICE_VSI_NETDEV_REGISTERED, vsi->state))) {
-		unregister_netdev(vsi->netdev);
-		clear_bit(ICE_VSI_NETDEV_REGISTERED, vsi->state);
-	}
-
-	if (vsi->type == ICE_VSI_PF)
-		ice_devlink_destroy_pf_port(pf);
-
 	if (test_bit(ICE_FLAG_RSS_ENA, pf->flags))
 		ice_rss_clean(vsi);
 
 	ice_vsi_close(vsi);
 	ice_vsi_decfg(vsi);
 
-	if (vsi->netdev) {
-		if (test_bit(ICE_VSI_NETDEV_REGISTERED, vsi->state)) {
-			unregister_netdev(vsi->netdev);
-			clear_bit(ICE_VSI_NETDEV_REGISTERED, vsi->state);
-		}
-		if (test_bit(ICE_VSI_NETDEV_ALLOCD, vsi->state)) {
-			free_netdev(vsi->netdev);
-			vsi->netdev = NULL;
-			clear_bit(ICE_VSI_NETDEV_ALLOCD, vsi->state);
-		}
-	}
-
 	/* retain SW VSI data structure since it is needed to unregister and
 	 * free VSI netdev when PF is not in reset recovery pending state,\
 	 * for ex: during rmmod.
-- 
2.41.0

Re: [PATCH net v2] ice: Unregister netdev and devlink_port only once

[Jiri Pirko]

Mon, Jun 19, 2023 at 12:58:13PM CEST, poros@redhat.com wrote:
>Since commit 6624e780a577fc ("ice: split ice_vsi_setup into smaller
>functions") ice_vsi_release does things twice. There is unregister
>netdev which is unregistered in ice_deinit_eth also.

You need to describe more relationship between ice_vsi_release() and
ice_deinit_eth(). From a quick look, I don't see that ice_deinit_eth()
is always called before/afeter ice_vsi_release().

Ice init/release flows are very hard to follow :/


>
>It also unregisters the devlink_port twice which is also unregistered
>in ice_deinit_eth(). This double deregistration is hidden because
>devl_port_unregister ignores the return value of xa_erase.

This call for another patch, doesn't it? :)


>
>[   68.642167] Call Trace:
>[   68.650385]  ice_devlink_destroy_pf_port+0xe/0x20 [ice]
>[   68.655656]  ice_vsi_release+0x445/0x690 [ice]
>[   68.660147]  ice_deinit+0x99/0x280 [ice]
>[   68.664117]  ice_remove+0x1b6/0x5c0 [ice]
>
>[  171.103841] Call Trace:
>[  171.109607]  ice_devlink_destroy_pf_port+0xf/0x20 [ice]
>[  171.114841]  ice_remove+0x158/0x270 [ice]
>[  171.118854]  pci_device_remove+0x3b/0xc0
>[  171.122779]  device_release_driver_internal+0xc7/0x170
>[  171.127912]  driver_detach+0x54/0x8c
>[  171.131491]  bus_remove_driver+0x77/0xd1
>[  171.135406]  pci_unregister_driver+0x2d/0xb0
>[  171.139670]  ice_module_exit+0xc/0x55f [ice]
>
>Fixes: 6624e780a577 ("ice: split ice_vsi_setup into smaller functions")
>Signed-off-by: Petr Oros <poros@redhat.com>
>---
>v2: reword subject
>
>v1: https://lore.kernel.org/netdev/20230619084948.360128-1-poros@redhat.com/
>---
> drivers/net/ethernet/intel/ice/ice_lib.c | 27 ------------------------
> 1 file changed, 27 deletions(-)
>
>diff --git a/drivers/net/ethernet/intel/ice/ice_lib.c b/drivers/net/ethernet/intel/ice/ice_lib.c
>index 11ae0e41f518a1..284a1f0bfdb545 100644
>--- a/drivers/net/ethernet/intel/ice/ice_lib.c
>+++ b/drivers/net/ethernet/intel/ice/ice_lib.c
>@@ -3272,39 +3272,12 @@ int ice_vsi_release(struct ice_vsi *vsi)
> 		return -ENODEV;
> 	pf = vsi->back;
> 
>-	/* do not unregister while driver is in the reset recovery pending
>-	 * state. Since reset/rebuild happens through PF service task workqueue,
>-	 * it's not a good idea to unregister netdev that is associated to the
>-	 * PF that is running the work queue items currently. This is done to
>-	 * avoid check_flush_dependency() warning on this wq
>-	 */
>-	if (vsi->netdev && !ice_is_reset_in_progress(pf->state) &&
>-	    (test_bit(ICE_VSI_NETDEV_REGISTERED, vsi->state))) {
>-		unregister_netdev(vsi->netdev);
>-		clear_bit(ICE_VSI_NETDEV_REGISTERED, vsi->state);
>-	}
>-
>-	if (vsi->type == ICE_VSI_PF)
>-		ice_devlink_destroy_pf_port(pf);
>-
> 	if (test_bit(ICE_FLAG_RSS_ENA, pf->flags))
> 		ice_rss_clean(vsi);
> 
> 	ice_vsi_close(vsi);
> 	ice_vsi_decfg(vsi);
> 
>-	if (vsi->netdev) {
>-		if (test_bit(ICE_VSI_NETDEV_REGISTERED, vsi->state)) {
>-			unregister_netdev(vsi->netdev);
>-			clear_bit(ICE_VSI_NETDEV_REGISTERED, vsi->state);
>-		}
>-		if (test_bit(ICE_VSI_NETDEV_ALLOCD, vsi->state)) {
>-			free_netdev(vsi->netdev);
>-			vsi->netdev = NULL;
>-			clear_bit(ICE_VSI_NETDEV_ALLOCD, vsi->state);
>-		}
>-	}
>-
> 	/* retain SW VSI data structure since it is needed to unregister and
> 	 * free VSI netdev when PF is not in reset recovery pending state,\
> 	 * for ex: during rmmod.
>-- 
>2.41.0
>
>

Re: [Intel-wired-lan] [PATCH net v2] ice: Unregister netdev and devlink_port only once

[Maciej Fijalkowski]

On Mon, Jun 19, 2023 at 12:58:13PM +0200, Petr Oros wrote:
> Since commit 6624e780a577fc ("ice: split ice_vsi_setup into smaller
> functions") ice_vsi_release does things twice. There is unregister
> netdev which is unregistered in ice_deinit_eth also.
> 
> It also unregisters the devlink_port twice which is also unregistered
> in ice_deinit_eth(). This double deregistration is hidden because
> devl_port_unregister ignores the return value of xa_erase.
> 
> [   68.642167] Call Trace:
> [   68.650385]  ice_devlink_destroy_pf_port+0xe/0x20 [ice]
> [   68.655656]  ice_vsi_release+0x445/0x690 [ice]
> [   68.660147]  ice_deinit+0x99/0x280 [ice]
> [   68.664117]  ice_remove+0x1b6/0x5c0 [ice]
> 
> [  171.103841] Call Trace:
> [  171.109607]  ice_devlink_destroy_pf_port+0xf/0x20 [ice]
> [  171.114841]  ice_remove+0x158/0x270 [ice]
> [  171.118854]  pci_device_remove+0x3b/0xc0
> [  171.122779]  device_release_driver_internal+0xc7/0x170
> [  171.127912]  driver_detach+0x54/0x8c
> [  171.131491]  bus_remove_driver+0x77/0xd1
> [  171.135406]  pci_unregister_driver+0x2d/0xb0
> [  171.139670]  ice_module_exit+0xc/0x55f [ice]
> 
> Fixes: 6624e780a577 ("ice: split ice_vsi_setup into smaller functions")
> Signed-off-by: Petr Oros <poros@redhat.com>
> ---
> v2: reword subject
> 
> v1: https://lore.kernel.org/netdev/20230619084948.360128-1-poros@redhat.com/
> ---
>  drivers/net/ethernet/intel/ice/ice_lib.c | 27 ------------------------
>  1 file changed, 27 deletions(-)
> 
> diff --git a/drivers/net/ethernet/intel/ice/ice_lib.c b/drivers/net/ethernet/intel/ice/ice_lib.c
> index 11ae0e41f518a1..284a1f0bfdb545 100644
> --- a/drivers/net/ethernet/intel/ice/ice_lib.c
> +++ b/drivers/net/ethernet/intel/ice/ice_lib.c
> @@ -3272,39 +3272,12 @@ int ice_vsi_release(struct ice_vsi *vsi)
>  		return -ENODEV;
>  	pf = vsi->back;
>  
> -	/* do not unregister while driver is in the reset recovery pending
> -	 * state. Since reset/rebuild happens through PF service task workqueue,
> -	 * it's not a good idea to unregister netdev that is associated to the
> -	 * PF that is running the work queue items currently. This is done to
> -	 * avoid check_flush_dependency() warning on this wq
> -	 */
> -	if (vsi->netdev && !ice_is_reset_in_progress(pf->state) &&
> -	    (test_bit(ICE_VSI_NETDEV_REGISTERED, vsi->state))) {
> -		unregister_netdev(vsi->netdev);
> -		clear_bit(ICE_VSI_NETDEV_REGISTERED, vsi->state);
> -	}
> -
> -	if (vsi->type == ICE_VSI_PF)
> -		ice_devlink_destroy_pf_port(pf);
> -
>  	if (test_bit(ICE_FLAG_RSS_ENA, pf->flags))
>  		ice_rss_clean(vsi);
>  
>  	ice_vsi_close(vsi);
>  	ice_vsi_decfg(vsi);
>  
> -	if (vsi->netdev) {
> -		if (test_bit(ICE_VSI_NETDEV_REGISTERED, vsi->state)) {
> -			unregister_netdev(vsi->netdev);
> -			clear_bit(ICE_VSI_NETDEV_REGISTERED, vsi->state);
> -		}
> -		if (test_bit(ICE_VSI_NETDEV_ALLOCD, vsi->state)) {
> -			free_netdev(vsi->netdev);
> -			vsi->netdev = NULL;
> -			clear_bit(ICE_VSI_NETDEV_ALLOCD, vsi->state);
> -		}
> -	}

Hmm it's a bit messy:D

I agree that ice_deinit_eth() should be the one that takes care of netdev
clean up as ice_init_eth() was the one to alloc and register it. I believe
that part of the split up work was to come up with entities that are
scoped to alloc/dealloc related resources...as this was even more messed
up before.

It would be worth to exercise other code paths where ice_vsi_release() is
used - like do a loopback test (ethtool -t $IFACE) or go through a reset.

Reviewed-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com>

> -
>  	/* retain SW VSI data structure since it is needed to unregister and
>  	 * free VSI netdev when PF is not in reset recovery pending state,\
>  	 * for ex: during rmmod.
> -- 
> 2.41.0
> 
> _______________________________________________
> Intel-wired-lan mailing list
> Intel-wired-lan@osuosl.org
> https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: [PATCH net v2] ice: Unregister netdev and devlink_port only once

[Maciej Fijalkowski]

On Mon, Jun 19, 2023 at 05:07:21PM +0200, Jiri Pirko wrote:
> Mon, Jun 19, 2023 at 12:58:13PM CEST, poros@redhat.com wrote:
> >Since commit 6624e780a577fc ("ice: split ice_vsi_setup into smaller
> >functions") ice_vsi_release does things twice. There is unregister
> >netdev which is unregistered in ice_deinit_eth also.
> 
> You need to describe more relationship between ice_vsi_release() and
> ice_deinit_eth(). From a quick look, I don't see that ice_deinit_eth()
> is always called before/afeter ice_vsi_release().
> 

Only relation between them is that they both are called on .remove() pci
callback. Clearly ice_vsi_release() should not be cleaning up stuff that
ice_init_eth() has brought up. I was staring at the code and jumping
around these flows and fix seems reasonable.

> Ice init/release flows are very hard to follow :/
> 
> 
> >
> >It also unregisters the devlink_port twice which is also unregistered
> >in ice_deinit_eth(). This double deregistration is hidden because
> >devl_port_unregister ignores the return value of xa_erase.
> 
> This call for another patch, doesn't it? :)

+1 :)

> 
> 
> >
> >[   68.642167] Call Trace:
> >[   68.650385]  ice_devlink_destroy_pf_port+0xe/0x20 [ice]
> >[   68.655656]  ice_vsi_release+0x445/0x690 [ice]
> >[   68.660147]  ice_deinit+0x99/0x280 [ice]
> >[   68.664117]  ice_remove+0x1b6/0x5c0 [ice]
> >
> >[  171.103841] Call Trace:
> >[  171.109607]  ice_devlink_destroy_pf_port+0xf/0x20 [ice]
> >[  171.114841]  ice_remove+0x158/0x270 [ice]
> >[  171.118854]  pci_device_remove+0x3b/0xc0
> >[  171.122779]  device_release_driver_internal+0xc7/0x170
> >[  171.127912]  driver_detach+0x54/0x8c
> >[  171.131491]  bus_remove_driver+0x77/0xd1
> >[  171.135406]  pci_unregister_driver+0x2d/0xb0
> >[  171.139670]  ice_module_exit+0xc/0x55f [ice]
> >
> >Fixes: 6624e780a577 ("ice: split ice_vsi_setup into smaller functions")
> >Signed-off-by: Petr Oros <poros@redhat.com>
> >---
> >v2: reword subject
> >
> >v1: https://lore.kernel.org/netdev/20230619084948.360128-1-poros@redhat.com/
> >---
> > drivers/net/ethernet/intel/ice/ice_lib.c | 27 ------------------------
> > 1 file changed, 27 deletions(-)
> >
> >diff --git a/drivers/net/ethernet/intel/ice/ice_lib.c b/drivers/net/ethernet/intel/ice/ice_lib.c
> >index 11ae0e41f518a1..284a1f0bfdb545 100644
> >--- a/drivers/net/ethernet/intel/ice/ice_lib.c
> >+++ b/drivers/net/ethernet/intel/ice/ice_lib.c
> >@@ -3272,39 +3272,12 @@ int ice_vsi_release(struct ice_vsi *vsi)
> > 		return -ENODEV;
> > 	pf = vsi->back;
> > 
> >-	/* do not unregister while driver is in the reset recovery pending
> >-	 * state. Since reset/rebuild happens through PF service task workqueue,
> >-	 * it's not a good idea to unregister netdev that is associated to the
> >-	 * PF that is running the work queue items currently. This is done to
> >-	 * avoid check_flush_dependency() warning on this wq
> >-	 */
> >-	if (vsi->netdev && !ice_is_reset_in_progress(pf->state) &&
> >-	    (test_bit(ICE_VSI_NETDEV_REGISTERED, vsi->state))) {
> >-		unregister_netdev(vsi->netdev);
> >-		clear_bit(ICE_VSI_NETDEV_REGISTERED, vsi->state);
> >-	}
> >-
> >-	if (vsi->type == ICE_VSI_PF)
> >-		ice_devlink_destroy_pf_port(pf);
> >-
> > 	if (test_bit(ICE_FLAG_RSS_ENA, pf->flags))
> > 		ice_rss_clean(vsi);
> > 
> > 	ice_vsi_close(vsi);
> > 	ice_vsi_decfg(vsi);
> > 
> >-	if (vsi->netdev) {
> >-		if (test_bit(ICE_VSI_NETDEV_REGISTERED, vsi->state)) {
> >-			unregister_netdev(vsi->netdev);
> >-			clear_bit(ICE_VSI_NETDEV_REGISTERED, vsi->state);
> >-		}
> >-		if (test_bit(ICE_VSI_NETDEV_ALLOCD, vsi->state)) {
> >-			free_netdev(vsi->netdev);
> >-			vsi->netdev = NULL;
> >-			clear_bit(ICE_VSI_NETDEV_ALLOCD, vsi->state);
> >-		}
> >-	}
> >-
> > 	/* retain SW VSI data structure since it is needed to unregister and
> > 	 * free VSI netdev when PF is not in reset recovery pending state,\
> > 	 * for ex: during rmmod.
> >-- 
> >2.41.0
> >
> >
> 

Re: [Intel-wired-lan] [PATCH net v2] ice: Unregister netdev and devlink_port only once

[Petr Oros]

Petr Oros píše v Po 19. 06. 2023 v 12:58 +0200:
> Since commit 6624e780a577fc ("ice: split ice_vsi_setup into smaller
> functions") ice_vsi_release does things twice. There is unregister
> netdev which is unregistered in ice_deinit_eth also.
> 
> It also unregisters the devlink_port twice which is also unregistered
> in ice_deinit_eth(). This double deregistration is hidden because
> devl_port_unregister ignores the return value of xa_erase.
> 
Hi,

Is it possible to push this patch forward? I think the questions have
been answered. Or is something still unclear?

Regards,
Petr

Re: [Intel-wired-lan] [PATCH net v2] ice: Unregister netdev and devlink_port only once

[Tony Nguyen]

On 7/6/2023 2:30 PM, Petr Oros wrote:
> Petr Oros píše v Po 19. 06. 2023 v 12:58 +0200:
>> Since commit 6624e780a577fc ("ice: split ice_vsi_setup into smaller
>> functions") ice_vsi_release does things twice. There is unregister
>> netdev which is unregistered in ice_deinit_eth also.
>>
>> It also unregisters the devlink_port twice which is also unregistered
>> in ice_deinit_eth(). This double deregistration is hidden because
>> devl_port_unregister ignores the return value of xa_erase.
>>
> Hi,
> 
> Is it possible to push this patch forward? I think the questions have
> been answered. Or is something still unclear?

I was expecting a new version based on the question/request [1] (so set 
to changes requested) before the response. Will apply and have it move 
forward.

Thanks,
Tony

[1] 
https://lore.kernel.org/intel-wired-lan/eda7b84e56123bce167a64133572440e6806ef1e.camel@redhat.com/T/#mbef0a70424d01bb8fcc1ae1a4865c617fbb4befc

RE: [Intel-wired-lan] [PATCH net v2] ice: Unregister netdev and devlink_port only once

[Pucha, HimasekharX Reddy]

> -----Original Message-----
> From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf Of Petr Oros
> Sent: Monday, June 19, 2023 4:28 PM
> To: netdev@vger.kernel.org
> Cc: pmenzel@molgen.mpg.de; intel-wired-lan@lists.osuosl.org; Brandeburg, Jesse <jesse.brandeburg@intel.com>; linux-kernel@vger.kernel.org; edumazet@google.com; Nguyen, Anthony L <anthony.l.nguyen@intel.com>; kuba@kernel.org; pabeni@redhat.com; davem@davemloft.net
> Subject: [Intel-wired-lan] [PATCH net v2] ice: Unregister netdev and devlink_port only once
>
> Since commit 6624e780a577fc ("ice: split ice_vsi_setup into smaller
> functions") ice_vsi_release does things twice. There is unregister netdev which is unregistered in ice_deinit_eth also.
>
> It also unregisters the devlink_port twice which is also unregistered in ice_deinit_eth(). This double deregistration is hidden because devl_port_unregister ignores the return value of xa_erase.
>
> [   68.642167] Call Trace:
> [   68.650385]  ice_devlink_destroy_pf_port+0xe/0x20 [ice]
> [   68.655656]  ice_vsi_release+0x445/0x690 [ice]
> [   68.660147]  ice_deinit+0x99/0x280 [ice]
> [   68.664117]  ice_remove+0x1b6/0x5c0 [ice]
>
> [  171.103841] Call Trace:
> [  171.109607]  ice_devlink_destroy_pf_port+0xf/0x20 [ice] [  171.114841]  ice_remove+0x158/0x270 [ice] [  171.118854]  pci_device_remove+0x3b/0xc0 [  171.122779]  device_release_driver_internal+0xc7/0x170
> [  171.127912]  driver_detach+0x54/0x8c
> [  171.131491]  bus_remove_driver+0x77/0xd1 [  171.135406]  pci_unregister_driver+0x2d/0xb0 [  171.139670]  ice_module_exit+0xc/0x55f [ice]
>
> Fixes: 6624e780a577 ("ice: split ice_vsi_setup into smaller functions")
> Signed-off-by: Petr Oros <poros@redhat.com>
> ---
> v2: reword subject
>
> v1: https://lore.kernel.org/netdev/20230619084948.360128-1-poros@redhat.com/
> ---
> drivers/net/ethernet/intel/ice/ice_lib.c | 27 ------------------------
> 1 file changed, 27 deletions(-)
>

Tested-by: Pucha Himasekhar Reddy <himasekharx.reddy.pucha@intel.com> (A Contingent worker at Intel)