[PATCH] net/mlx4: Fix EEH recovery failure

[PATCH] net/mlx4: Fix EEH recovery failure

[Gavin Shan]

The patch fixes couple of EEH recovery failures on PPC PowerNV
platform:

   * Release reserved memory regions in mlx4_pci_err_detected().
     Otherwise, __mlx4_init_one() fails because of reserving
     same memory regions recursively.
   * Disable PCI device in mlx4_pci_err_detected(). Otherwise,
     pci_enable_device() in __mlx4_init_one() doesn't enable
     the PCI device because it's already in enabled state indicated
     by struct pci_dev::enable_cnt.
   * Don't clear struct mlx4_priv instance in mlx4_pci_err_detected().
     Otherwise, __mlx4_init_one() runs into kernel crash because
     of dereferencing to NULL pointer.

With the patch applied, EEH recovery for mlx4 adapter succeeds on PPC
PowerNV platform.

   # lspci
   0003:0f:00.0 Network controller: Mellanox Technologies \
   MT27500 Family [ConnectX-3]

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
 drivers/net/ethernet/mellanox/mlx4/main.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c
index 90de6e1..e118ac9 100644
--- a/drivers/net/ethernet/mellanox/mlx4/main.c
+++ b/drivers/net/ethernet/mellanox/mlx4/main.c
@@ -2809,7 +2809,6 @@ static void mlx4_unload_one(struct pci_dev *pdev)
 	kfree(dev->caps.qp1_proxy);
 	kfree(dev->dev_vfs);
 
-	memset(priv, 0, sizeof(*priv));
 	priv->pci_dev_data = pci_dev_data;
 	priv->removed = 1;
 }
@@ -2900,6 +2899,8 @@ static pci_ers_result_t mlx4_pci_err_detected(struct pci_dev *pdev,
 					      pci_channel_state_t state)
 {
 	mlx4_unload_one(pdev);
+	pci_release_regions(pdev);
+	pci_disable_device(pdev);
 
 	return state == pci_channel_io_perm_failure ?
 		PCI_ERS_RESULT_DISCONNECT : PCI_ERS_RESULT_NEED_RESET;
-- 
1.8.3.2

Re: [PATCH] net/mlx4: Fix EEH recovery failure

[Amir Vadai]

On 11/22/2014 12:56 PM, Gavin Shan wrote:
> The patch fixes couple of EEH recovery failures on PPC PowerNV
> platform:
> 
>    * Release reserved memory regions in mlx4_pci_err_detected().
>      Otherwise, __mlx4_init_one() fails because of reserving
>      same memory regions recursively.
>    * Disable PCI device in mlx4_pci_err_detected(). Otherwise,
>      pci_enable_device() in __mlx4_init_one() doesn't enable
>      the PCI device because it's already in enabled state indicated
>      by struct pci_dev::enable_cnt.
>    * Don't clear struct mlx4_priv instance in mlx4_pci_err_detected().
>      Otherwise, __mlx4_init_one() runs into kernel crash because
>      of dereferencing to NULL pointer.
> 
> With the patch applied, EEH recovery for mlx4 adapter succeeds on PPC
> PowerNV platform.
> 
>    # lspci
>    0003:0f:00.0 Network controller: Mellanox Technologies \
>    MT27500 Family [ConnectX-3]
> 
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>

Hi Gavin,

Yishai (added to the CC) is few days before sending a patchset to fix
the reset flow and inside it there is a fix to EEH recovery.
I would be happy if you could wait for the whole reset flow fix by Yishai.

If you'd like, I can send you the patchset to try. Currently it is under
review inside Mellanox before being sent to the mailing list.

Thanks,
Amir

Re: [PATCH] net/mlx4: Fix EEH recovery failure

[Gavin Shan]

On Sun, Nov 23, 2014 at 06:21:47PM +0200, Amir Vadai wrote:
>On 11/22/2014 12:56 PM, Gavin Shan wrote:
>> The patch fixes couple of EEH recovery failures on PPC PowerNV
>> platform:
>> 
>>    * Release reserved memory regions in mlx4_pci_err_detected().
>>      Otherwise, __mlx4_init_one() fails because of reserving
>>      same memory regions recursively.
>>    * Disable PCI device in mlx4_pci_err_detected(). Otherwise,
>>      pci_enable_device() in __mlx4_init_one() doesn't enable
>>      the PCI device because it's already in enabled state indicated
>>      by struct pci_dev::enable_cnt.
>>    * Don't clear struct mlx4_priv instance in mlx4_pci_err_detected().
>>      Otherwise, __mlx4_init_one() runs into kernel crash because
>>      of dereferencing to NULL pointer.
>> 
>> With the patch applied, EEH recovery for mlx4 adapter succeeds on PPC
>> PowerNV platform.
>> 
>>    # lspci
>>    0003:0f:00.0 Network controller: Mellanox Technologies \
>>    MT27500 Family [ConnectX-3]
>> 
>> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>
>Hi Gavin,
>
>Yishai (added to the CC) is few days before sending a patchset to fix
>the reset flow and inside it there is a fix to EEH recovery.
>I would be happy if you could wait for the whole reset flow fix by Yishai.
>

Yes, It's not urgent and I can wait. Thanks for the info.

>If you'd like, I can send you the patchset to try. Currently it is under
>review inside Mellanox before being sent to the mailing list.
>

It would be nice to send me the patchset for me to have a try.

Thanks,
Gavin

>Thanks,
>Amir
>
>
>
>

Re: [PATCH] net/mlx4: Fix EEH recovery failure

[Gavin Shan]

On Sat, Nov 22, 2014 at 09:56:47PM +1100, Gavin Shan wrote:

Yishai already had patches fixing the issue. So please ignore
this patch and drop it.

Thanks,
Gavin

>The patch fixes couple of EEH recovery failures on PPC PowerNV
>platform:
>
>   * Release reserved memory regions in mlx4_pci_err_detected().
>     Otherwise, __mlx4_init_one() fails because of reserving
>     same memory regions recursively.
>   * Disable PCI device in mlx4_pci_err_detected(). Otherwise,
>     pci_enable_device() in __mlx4_init_one() doesn't enable
>     the PCI device because it's already in enabled state indicated
>     by struct pci_dev::enable_cnt.
>   * Don't clear struct mlx4_priv instance in mlx4_pci_err_detected().
>     Otherwise, __mlx4_init_one() runs into kernel crash because
>     of dereferencing to NULL pointer.
>
>With the patch applied, EEH recovery for mlx4 adapter succeeds on PPC
>PowerNV platform.
>
>   # lspci
>   0003:0f:00.0 Network controller: Mellanox Technologies \
>   MT27500 Family [ConnectX-3]
>
>Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>---
> drivers/net/ethernet/mellanox/mlx4/main.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
>diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c
>index 90de6e1..e118ac9 100644
>--- a/drivers/net/ethernet/mellanox/mlx4/main.c
>+++ b/drivers/net/ethernet/mellanox/mlx4/main.c
>@@ -2809,7 +2809,6 @@ static void mlx4_unload_one(struct pci_dev *pdev)
> 	kfree(dev->caps.qp1_proxy);
> 	kfree(dev->dev_vfs);
>
>-	memset(priv, 0, sizeof(*priv));
> 	priv->pci_dev_data = pci_dev_data;
> 	priv->removed = 1;
> }
>@@ -2900,6 +2899,8 @@ static pci_ers_result_t mlx4_pci_err_detected(struct pci_dev *pdev,
> 					      pci_channel_state_t state)
> {
> 	mlx4_unload_one(pdev);
>+	pci_release_regions(pdev);
>+	pci_disable_device(pdev);
>
> 	return state == pci_channel_io_perm_failure ?
> 		PCI_ERS_RESULT_DISCONNECT : PCI_ERS_RESULT_NEED_RESET;
>-- 
>1.8.3.2
>

Re: [PATCH] net/mlx4: Fix EEH recovery failure

[Or Gerlitz]

On Sat, Nov 22, 2014 at 12:56 PM, Gavin Shan <gwshan@linux.vnet.ibm.com> wrote:
> The patch fixes couple of EEH recovery failures on PPC PowerNV
> platform:

>    * Don't clear struct mlx4_priv instance in mlx4_pci_err_detected().
>      Otherwise, __mlx4_init_one() runs into kernel crash because
>      of dereferencing to NULL pointer.

I don't see this change in the patch, I see no-clearing of mlx4_priv
in __mlx4_unload_one - please clarify, also is this patch
based/targeted on the net or net-next tree?



> With the patch applied, EEH recovery for mlx4 adapter succeeds on PPC
> PowerNV platform.
>
>    # lspci
>    0003:0f:00.0 Network controller: Mellanox Technologies \
>    MT27500 Family [ConnectX-3]
>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> ---
>  drivers/net/ethernet/mellanox/mlx4/main.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c
> index 90de6e1..e118ac9 100644
> --- a/drivers/net/ethernet/mellanox/mlx4/main.c
> +++ b/drivers/net/ethernet/mellanox/mlx4/main.c
> @@ -2809,7 +2809,6 @@ static void mlx4_unload_one(struct pci_dev *pdev)
>         kfree(dev->caps.qp1_proxy);
>         kfree(dev->dev_vfs);
>
> -       memset(priv, 0, sizeof(*priv));
>         priv->pci_dev_data = pci_dev_data;
>         priv->removed = 1;
>  }
> @@ -2900,6 +2899,8 @@ static pci_ers_result_t mlx4_pci_err_detected(struct pci_dev *pdev,
>                                               pci_channel_state_t state)
>  {
>         mlx4_unload_one(pdev);
> +       pci_release_regions(pdev);
> +       pci_disable_device(pdev);
>
>         return state == pci_channel_io_perm_failure ?
>                 PCI_ERS_RESULT_DISCONNECT : PCI_ERS_RESULT_NEED_RESET;
> --
> 1.8.3.2
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] net/mlx4: Fix EEH recovery failure

[Gavin Shan]

On Mon, Nov 24, 2014 at 11:17:55PM +0200, Or Gerlitz wrote:
>On Sat, Nov 22, 2014 at 12:56 PM, Gavin Shan <gwshan@linux.vnet.ibm.com> wrote:
>> The patch fixes couple of EEH recovery failures on PPC PowerNV
>> platform:
>
>>    * Don't clear struct mlx4_priv instance in mlx4_pci_err_detected().
>>      Otherwise, __mlx4_init_one() runs into kernel crash because
>>      of dereferencing to NULL pointer.
>
>I don't see this change in the patch, I see no-clearing of mlx4_priv
>in __mlx4_unload_one - please clarify, also is this patch
>based/targeted on the net or net-next tree?
>

Yes, It would be: Don't clear struct mlx4_priv instance in mlx4_unload_one(),
which is called by mlx4_pci_err_detected().

It's based on 3.18.rc5, where I had couple of EEH fixes on top of it.
When testing EEH with it, I hit the issue.

Thanks,
Gavin

>
>
>> With the patch applied, EEH recovery for mlx4 adapter succeeds on PPC
>> PowerNV platform.
>>
>>    # lspci
>>    0003:0f:00.0 Network controller: Mellanox Technologies \
>>    MT27500 Family [ConnectX-3]
>>
>> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>> ---
>>  drivers/net/ethernet/mellanox/mlx4/main.c | 3 ++-
>>  1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c
>> index 90de6e1..e118ac9 100644
>> --- a/drivers/net/ethernet/mellanox/mlx4/main.c
>> +++ b/drivers/net/ethernet/mellanox/mlx4/main.c
>> @@ -2809,7 +2809,6 @@ static void mlx4_unload_one(struct pci_dev *pdev)
>>         kfree(dev->caps.qp1_proxy);
>>         kfree(dev->dev_vfs);
>>
>> -       memset(priv, 0, sizeof(*priv));
>>         priv->pci_dev_data = pci_dev_data;
>>         priv->removed = 1;
>>  }
>> @@ -2900,6 +2899,8 @@ static pci_ers_result_t mlx4_pci_err_detected(struct pci_dev *pdev,
>>                                               pci_channel_state_t state)
>>  {
>>         mlx4_unload_one(pdev);
>> +       pci_release_regions(pdev);
>> +       pci_disable_device(pdev);
>>
>>         return state == pci_channel_io_perm_failure ?
>>                 PCI_ERS_RESULT_DISCONNECT : PCI_ERS_RESULT_NEED_RESET;
>> --
>> 1.8.3.2
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [PATCH] net/mlx4: Fix EEH recovery failure

[Or Gerlitz]

On Mon, Nov 24, 2014 at 11:55 PM, Gavin Shan <gwshan@linux.vnet.ibm.com> wrote:
> On Mon, Nov 24, 2014 at 11:17:55PM +0200, Or Gerlitz wrote:
>>On Sat, Nov 22, 2014 at 12:56 PM, Gavin Shan <gwshan@linux.vnet.ibm.com> wrote:
>>> The patch fixes couple of EEH recovery failures on PPC PowerNV
>>> platform:
>>
>>>    * Don't clear struct mlx4_priv instance in mlx4_pci_err_detected().
>>>      Otherwise, __mlx4_init_one() runs into kernel crash because
>>>      of dereferencing to NULL pointer.
>>
>>I don't see this change in the patch, I see no-clearing of mlx4_priv
>>in __mlx4_unload_one - please clarify, also is this patch
>>based/targeted on the net or net-next tree?
>>
>
> Yes, It would be: Don't clear struct mlx4_priv instance in mlx4_unload_one(),
> which is called by mlx4_pci_err_detected().


But the struct mlx4_priv instance is cleared in mlx4_unload_one() for
a reason, I suspect that you might made the EEH callback to work, but
broke something else... e.g did you made sure that kexec works after
your changes as it did before?

> It's based on 3.18.rc5, where I had couple of EEH fixes on top of it.
> When testing EEH with it, I hit the issue.

>>> With the patch applied, EEH recovery for mlx4 adapter succeeds on PPC
>>> PowerNV platform.
>>>
>>>    # lspci
>>>    0003:0f:00.0 Network controller: Mellanox Technologies \
>>>    MT27500 Family [ConnectX-3]
>>>
>>> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>>> ---
>>>  drivers/net/ethernet/mellanox/mlx4/main.c | 3 ++-
>>>  1 file changed, 2 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c
>>> index 90de6e1..e118ac9 100644
>>> --- a/drivers/net/ethernet/mellanox/mlx4/main.c
>>> +++ b/drivers/net/ethernet/mellanox/mlx4/main.c
>>> @@ -2809,7 +2809,6 @@ static void mlx4_unload_one(struct pci_dev *pdev)
>>>         kfree(dev->caps.qp1_proxy);
>>>         kfree(dev->dev_vfs);
>>>
>>> -       memset(priv, 0, sizeof(*priv));
>>>         priv->pci_dev_data = pci_dev_data;
>>>         priv->removed = 1;
>>>  }
>>> @@ -2900,6 +2899,8 @@ static pci_ers_result_t mlx4_pci_err_detected(struct pci_dev *pdev,
>>>                                               pci_channel_state_t state)
>>>  {
>>>         mlx4_unload_one(pdev);
>>> +       pci_release_regions(pdev);
>>> +       pci_disable_device(pdev);
>>>
>>>         return state == pci_channel_io_perm_failure ?
>>>                 PCI_ERS_RESULT_DISCONNECT : PCI_ERS_RESULT_NEED_RESET;
>>> --

Re: [PATCH] net/mlx4: Fix EEH recovery failure

[Gavin Shan]

On Wed, Nov 26, 2014 at 12:00:31AM +0200, Or Gerlitz wrote:
>On Mon, Nov 24, 2014 at 11:55 PM, Gavin Shan <gwshan@linux.vnet.ibm.com> wrote:
>> On Mon, Nov 24, 2014 at 11:17:55PM +0200, Or Gerlitz wrote:
>>>On Sat, Nov 22, 2014 at 12:56 PM, Gavin Shan <gwshan@linux.vnet.ibm.com> wrote:
>>>> The patch fixes couple of EEH recovery failures on PPC PowerNV
>>>> platform:
>>>
>>>>    * Don't clear struct mlx4_priv instance in mlx4_pci_err_detected().
>>>>      Otherwise, __mlx4_init_one() runs into kernel crash because
>>>>      of dereferencing to NULL pointer.
>>>
>>>I don't see this change in the patch, I see no-clearing of mlx4_priv
>>>in __mlx4_unload_one - please clarify, also is this patch
>>>based/targeted on the net or net-next tree?
>>>
>>
>> Yes, It would be: Don't clear struct mlx4_priv instance in mlx4_unload_one(),
>> which is called by mlx4_pci_err_detected().
>
>
>But the struct mlx4_priv instance is cleared in mlx4_unload_one() for
>a reason, I suspect that you might made the EEH callback to work, but
>broke something else... e.g did you made sure that kexec works after
>your changes as it did before?
>

Nope, I didn't try kexec out and I'll have a try, thanks!

Gavin

>> It's based on 3.18.rc5, where I had couple of EEH fixes on top of it.
>> When testing EEH with it, I hit the issue.
>
>>>> With the patch applied, EEH recovery for mlx4 adapter succeeds on PPC
>>>> PowerNV platform.
>>>>
>>>>    # lspci
>>>>    0003:0f:00.0 Network controller: Mellanox Technologies \
>>>>    MT27500 Family [ConnectX-3]
>>>>
>>>> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>>>> ---
>>>>  drivers/net/ethernet/mellanox/mlx4/main.c | 3 ++-
>>>>  1 file changed, 2 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c
>>>> index 90de6e1..e118ac9 100644
>>>> --- a/drivers/net/ethernet/mellanox/mlx4/main.c
>>>> +++ b/drivers/net/ethernet/mellanox/mlx4/main.c
>>>> @@ -2809,7 +2809,6 @@ static void mlx4_unload_one(struct pci_dev *pdev)
>>>>         kfree(dev->caps.qp1_proxy);
>>>>         kfree(dev->dev_vfs);
>>>>
>>>> -       memset(priv, 0, sizeof(*priv));
>>>>         priv->pci_dev_data = pci_dev_data;
>>>>         priv->removed = 1;
>>>>  }
>>>> @@ -2900,6 +2899,8 @@ static pci_ers_result_t mlx4_pci_err_detected(struct pci_dev *pdev,
>>>>                                               pci_channel_state_t state)
>>>>  {
>>>>         mlx4_unload_one(pdev);
>>>> +       pci_release_regions(pdev);
>>>> +       pci_disable_device(pdev);
>>>>
>>>>         return state == pci_channel_io_perm_failure ?
>>>>                 PCI_ERS_RESULT_DISCONNECT : PCI_ERS_RESULT_NEED_RESET;
>>>> --
>