[PATCH net-next 3/3] mlxsw: pci: Lock configuration space of upstream bridge during reset

[PATCH net-next 3/3] mlxsw: pci: Lock configuration space of upstream bridge during reset

[Petr Machata]

From: Ido Schimmel <idosch@nvidia.com>

The driver triggers a "Secondary Bus Reset" (SBR) by calling
__pci_reset_function_locked() which asserts the SBR bit in the "Bridge
Control Register" in the configuration space of the upstream bridge for
2ms. This is done without locking the configuration space of the
upstream bridge port, allowing user space to access it concurrently.

Linux 6.11 will start warning about such unlocked resets [1][2]:

pcieport 0000:00:01.0: unlocked secondary bus reset via: pci_reset_bus_function+0x51c/0x6a0

Avoid the warning by locking the configuration space of the upstream
bridge prior to the reset and unlocking it afterwards.

[1] https://lore.kernel.org/all/171711746953.1628941.4692125082286867825.stgit@dwillia2-xfh.jf.intel.com/
[2] https://lore.kernel.org/all/20240531213150.GA610983@bhelgaas/

Cc: linux-pci@vger.kernel.org
Signed-off-by: Ido Schimmel <idosch@nvidia.com>
Signed-off-by: Petr Machata <petrm@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlxsw/pci.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlxsw/pci.c b/drivers/net/ethernet/mellanox/mlxsw/pci.c
index 0320dabd1380..060e5b939211 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/pci.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/pci.c
@@ -1784,6 +1784,7 @@ static int mlxsw_pci_reset_at_pci_disable(struct mlxsw_pci *mlxsw_pci,
 {
 	struct pci_dev *pdev = mlxsw_pci->pdev;
 	char mrsr_pl[MLXSW_REG_MRSR_LEN];
+	struct pci_dev *bridge;
 	int err;
 
 	if (!pci_reset_sbr_supported) {
@@ -1800,6 +1801,9 @@ static int mlxsw_pci_reset_at_pci_disable(struct mlxsw_pci *mlxsw_pci,
 sbr:
 	device_lock_assert(&pdev->dev);
 
+	bridge = pci_upstream_bridge(pdev);
+	if (bridge)
+		pci_cfg_access_lock(bridge);
 	pci_cfg_access_lock(pdev);
 	pci_save_state(pdev);
 
@@ -1809,6 +1813,8 @@ static int mlxsw_pci_reset_at_pci_disable(struct mlxsw_pci *mlxsw_pci,
 
 	pci_restore_state(pdev);
 	pci_cfg_access_unlock(pdev);
+	if (bridge)
+		pci_cfg_access_unlock(bridge);
 
 	return err;
 }
-- 
2.45.0

Re: [PATCH net-next 3/3] mlxsw: pci: Lock configuration space of upstream bridge during reset

[Przemek Kitszel]

On 7/1/24 18:41, Petr Machata wrote:
> From: Ido Schimmel <idosch@nvidia.com>
> 
> The driver triggers a "Secondary Bus Reset" (SBR) by calling
> __pci_reset_function_locked() which asserts the SBR bit in the "Bridge
> Control Register" in the configuration space of the upstream bridge for
> 2ms. This is done without locking the configuration space of the
> upstream bridge port, allowing user space to access it concurrently.

This means your patch is a bugfix.

> 
> Linux 6.11 will start warning about such unlocked resets [1][2]:
> 
> pcieport 0000:00:01.0: unlocked secondary bus reset via: pci_reset_bus_function+0x51c/0x6a0
> 
> Avoid the warning by locking the configuration space of the upstream
> bridge prior to the reset and unlocking it afterwards.

You are not avoiding the warning but protecting concurrent access,
please add a Fixes tag.

> 
> [1] https://lore.kernel.org/all/171711746953.1628941.4692125082286867825.stgit@dwillia2-xfh.jf.intel.com/
> [2] https://lore.kernel.org/all/20240531213150.GA610983@bhelgaas/
> 
> Cc: linux-pci@vger.kernel.org
> Signed-off-by: Ido Schimmel <idosch@nvidia.com>
> Signed-off-by: Petr Machata <petrm@nvidia.com>
> ---
>   drivers/net/ethernet/mellanox/mlxsw/pci.c | 6 ++++++
>   1 file changed, 6 insertions(+)
> 
> diff --git a/drivers/net/ethernet/mellanox/mlxsw/pci.c b/drivers/net/ethernet/mellanox/mlxsw/pci.c
> index 0320dabd1380..060e5b939211 100644
> --- a/drivers/net/ethernet/mellanox/mlxsw/pci.c
> +++ b/drivers/net/ethernet/mellanox/mlxsw/pci.c
> @@ -1784,6 +1784,7 @@ static int mlxsw_pci_reset_at_pci_disable(struct mlxsw_pci *mlxsw_pci,
>   {
>   	struct pci_dev *pdev = mlxsw_pci->pdev;
>   	char mrsr_pl[MLXSW_REG_MRSR_LEN];
> +	struct pci_dev *bridge;
>   	int err;
>   
>   	if (!pci_reset_sbr_supported) {
> @@ -1800,6 +1801,9 @@ static int mlxsw_pci_reset_at_pci_disable(struct mlxsw_pci *mlxsw_pci,
>   sbr:
>   	device_lock_assert(&pdev->dev);
>   
> +	bridge = pci_upstream_bridge(pdev);
> +	if (bridge)
> +		pci_cfg_access_lock(bridge);
>   	pci_cfg_access_lock(pdev);
>   	pci_save_state(pdev);
>   
> @@ -1809,6 +1813,8 @@ static int mlxsw_pci_reset_at_pci_disable(struct mlxsw_pci *mlxsw_pci,
>   
>   	pci_restore_state(pdev);
>   	pci_cfg_access_unlock(pdev);
> +	if (bridge)
> +		pci_cfg_access_unlock(bridge);
>   
>   	return err;
>   }

Re: [PATCH net-next 3/3] mlxsw: pci: Lock configuration space of upstream bridge during reset

[Ido Schimmel]

On Tue, Jul 02, 2024 at 09:35:50AM +0200, Przemek Kitszel wrote:
> On 7/1/24 18:41, Petr Machata wrote:
> > From: Ido Schimmel <idosch@nvidia.com>
> > 
> > The driver triggers a "Secondary Bus Reset" (SBR) by calling
> > __pci_reset_function_locked() which asserts the SBR bit in the "Bridge
> > Control Register" in the configuration space of the upstream bridge for
> > 2ms. This is done without locking the configuration space of the
> > upstream bridge port, allowing user space to access it concurrently.
> 
> This means your patch is a bugfix.
> 
> > 
> > Linux 6.11 will start warning about such unlocked resets [1][2]:
> > 
> > pcieport 0000:00:01.0: unlocked secondary bus reset via: pci_reset_bus_function+0x51c/0x6a0
> > 
> > Avoid the warning by locking the configuration space of the upstream
> > bridge prior to the reset and unlocking it afterwards.
> 
> You are not avoiding the warning but protecting concurrent access,
> please add a Fixes tag.

The patch that added the missing lock in PCI core was posted without a
Fixes tag and merged as part of the 6.10 PR. See commit 7e89efc6e9e4
("PCI: Lock upstream bridge for pci_reset_function()").

I don't see a good reason for root to poke in the configuration space of
the upstream bridge during SBR, but AFAICT the worst that can happen is
that reset will fail and while it is a bug, it is not a regression.

Bjorn, do you see a reason to post this as a fix?

Thanks

> 
> > 
> > [1] https://lore.kernel.org/all/171711746953.1628941.4692125082286867825.stgit@dwillia2-xfh.jf.intel.com/
> > [2] https://lore.kernel.org/all/20240531213150.GA610983@bhelgaas/
> > 
> > Cc: linux-pci@vger.kernel.org
> > Signed-off-by: Ido Schimmel <idosch@nvidia.com>
> > Signed-off-by: Petr Machata <petrm@nvidia.com>

Re: [PATCH net-next 3/3] mlxsw: pci: Lock configuration space of upstream bridge during reset

[Bjorn Helgaas]

[+cc Dan]

On Wed, Jul 03, 2024 at 05:42:05PM +0300, Ido Schimmel wrote:
> On Tue, Jul 02, 2024 at 09:35:50AM +0200, Przemek Kitszel wrote:
> > On 7/1/24 18:41, Petr Machata wrote:
> > > From: Ido Schimmel <idosch@nvidia.com>
> > > 
> > > The driver triggers a "Secondary Bus Reset" (SBR) by calling
> > > __pci_reset_function_locked() which asserts the SBR bit in the "Bridge
> > > Control Register" in the configuration space of the upstream bridge for
> > > 2ms. This is done without locking the configuration space of the
> > > upstream bridge port, allowing user space to access it concurrently.
> > 
> > This means your patch is a bugfix.
> > 
> > > Linux 6.11 will start warning about such unlocked resets [1][2]:
> > > 
> > > pcieport 0000:00:01.0: unlocked secondary bus reset via: pci_reset_bus_function+0x51c/0x6a0
> > > 
> > > Avoid the warning by locking the configuration space of the upstream
> > > bridge prior to the reset and unlocking it afterwards.
> > 
> > You are not avoiding the warning but protecting concurrent access,
> > please add a Fixes tag.
> 
> The patch that added the missing lock in PCI core was posted without a
> Fixes tag and merged as part of the 6.10 PR. See commit 7e89efc6e9e4
> ("PCI: Lock upstream bridge for pci_reset_function()").
> 
> I don't see a good reason for root to poke in the configuration space of
> the upstream bridge during SBR, but AFAICT the worst that can happen is
> that reset will fail and while it is a bug, it is not a regression.
> 
> Bjorn, do you see a reason to post this as a fix?

Sorry, I was on vacation and missed this when I returned.

mlxsw is one of the few users of __pci_reset_function_locked().
Others are liquidio (octeon), VFIO, and Xen.

You need __pci_reset_function_locked() if you're already holding the
device mutex, i.e., device_lock(&pdev->dev).  I looked at the
mlxsw_pci_reset_at_pci_disable() path, and didn't see where it holds
that device lock, but I probably missed it.

The usual pci_reset_function() path, which would be preferable if you
can use it, does basically this:

  pci_dev_lock(bridge)
    device_lock(&bridge->dev)
    pci_cfg_access_lock(bridge)
  pci_dev_lock(pdev)
    device_lock(&pdev->dev)
    pci_cfg_access_lock(pdev)
  pci_dev_save_and_disable(dev)
  __pci_reset_function_locked(pdev)

This patch adds pci_cfg_access_lock(bridge), but doesn't acquire the
device_lock for the bridge.

It looks like you always reset the device at mlxsw_pci_probe()-time,
which is quite unusual in the first place, but I suppose there's some
good reason for it.

If you can use pci_reset_function() directly (or avoid the reset
altogether), it would be far preferable and would avoid potential
issues like the warning here.

Bjorn

> > > [1] https://lore.kernel.org/all/171711746953.1628941.4692125082286867825.stgit@dwillia2-xfh.jf.intel.com/
> > > [2] https://lore.kernel.org/all/20240531213150.GA610983@bhelgaas/
> > > 
> > > Cc: linux-pci@vger.kernel.org
> > > Signed-off-by: Ido Schimmel <idosch@nvidia.com>
> > > Signed-off-by: Petr Machata <petrm@nvidia.com>

Re: [PATCH net-next 3/3] mlxsw: pci: Lock configuration space of upstream bridge during reset

[Ido Schimmel]

Hi Bjorn,

On Fri, Jul 12, 2024 at 04:21:57PM -0500, Bjorn Helgaas wrote:
> [+cc Dan]
> 
> On Wed, Jul 03, 2024 at 05:42:05PM +0300, Ido Schimmel wrote:
> > On Tue, Jul 02, 2024 at 09:35:50AM +0200, Przemek Kitszel wrote:
> > > On 7/1/24 18:41, Petr Machata wrote:
> > > > From: Ido Schimmel <idosch@nvidia.com>
> > > > 
> > > > The driver triggers a "Secondary Bus Reset" (SBR) by calling
> > > > __pci_reset_function_locked() which asserts the SBR bit in the "Bridge
> > > > Control Register" in the configuration space of the upstream bridge for
> > > > 2ms. This is done without locking the configuration space of the
> > > > upstream bridge port, allowing user space to access it concurrently.
> > > 
> > > This means your patch is a bugfix.
> > > 
> > > > Linux 6.11 will start warning about such unlocked resets [1][2]:
> > > > 
> > > > pcieport 0000:00:01.0: unlocked secondary bus reset via: pci_reset_bus_function+0x51c/0x6a0
> > > > 
> > > > Avoid the warning by locking the configuration space of the upstream
> > > > bridge prior to the reset and unlocking it afterwards.
> > > 
> > > You are not avoiding the warning but protecting concurrent access,
> > > please add a Fixes tag.
> > 
> > The patch that added the missing lock in PCI core was posted without a
> > Fixes tag and merged as part of the 6.10 PR. See commit 7e89efc6e9e4
> > ("PCI: Lock upstream bridge for pci_reset_function()").
> > 
> > I don't see a good reason for root to poke in the configuration space of
> > the upstream bridge during SBR, but AFAICT the worst that can happen is
> > that reset will fail and while it is a bug, it is not a regression.
> > 
> > Bjorn, do you see a reason to post this as a fix?
> 
> Sorry, I was on vacation and missed this when I returned.
> 
> mlxsw is one of the few users of __pci_reset_function_locked().
> Others are liquidio (octeon), VFIO, and Xen.
> 
> You need __pci_reset_function_locked() if you're already holding the
> device mutex, i.e., device_lock(&pdev->dev).  I looked at the
> mlxsw_pci_reset_at_pci_disable() path, and didn't see where it holds
> that device lock, but I probably missed it.

It is locked. There is device_lock_assert(&pdev->dev) before the call to
__pci_reset_function_locked(pdev) to make sure this is the case.

> The usual pci_reset_function() path, which would be preferable if you
> can use it, does basically this:
> 
>   pci_dev_lock(bridge)
>     device_lock(&bridge->dev)
>     pci_cfg_access_lock(bridge)
>   pci_dev_lock(pdev)
>     device_lock(&pdev->dev)
>     pci_cfg_access_lock(pdev)
>   pci_dev_save_and_disable(dev)
>   __pci_reset_function_locked(pdev)
> 
> This patch adds pci_cfg_access_lock(bridge), but doesn't acquire the
> device_lock for the bridge.
> 
> It looks like you always reset the device at mlxsw_pci_probe()-time,
> which is quite unusual in the first place, but I suppose there's some
> good reason for it.

The driver resets the device to bring it to a known and clean state.
Older devices can be reset using a firmware command, but current
generation requires a PCI reset.

> If you can use pci_reset_function() directly (or avoid the reset
> altogether), it would be far preferable and would avoid potential
> issues like the warning here.

We couldn't use pci_reset_function() even if we didn't reset during
probe. Another call path that triggers the reset is "devlink reload"
which holds the devlink instance lock. Trying to acquire the device lock
while holding the devlink instance lock would result in lock inversion.
We modified devlink to acquire the device lock before the instance lock
so that we could call PCI APIs with the device lock held. See:

https://lore.kernel.org/netdev/20231017074257.3389177-1-idosch@nvidia.com/