Re: [PATCH RESEND] nvme-pci: Fix EEH failure on ppc after subsystem reset

[Nilay Shroff <nilay@linux.ibm.com>]

Hi,

A gentle ping...

Thanks,
--Nilay

On 2/16/24 18:07, Srimannarayana Murthy Maram wrote:
> Hi all,
> 
> Tested patch with upstream kernel "6.8.0-rc4"
> 
> Issue verified on IBM power systems with manual test case has following steps
> 1. Note nvme controllers for a nvme subsystem using
>     "nvme list-subsys"
> 2. Perform nvme subsystem reset on each listed controller
>    under nvme subsystem one after the other, only after successful recovery.
> 
> Verified it on power system with NVME device on normal and multipath (2 paths) configuration.
> 
> Provided patch successfully recovered single controller(normal) and both controller(multipath) listed under nvme subsystem.
> 
> Tested-by: Maram Srimannarayana Murthy<msmurthy@linux.vnet.ibm.com>
> 
> Thank you,
> Maram Srimannarayana Murthy
> Sr. Test Engineer | IBM
> 
> On 2/9/24 10:32, Nilay Shroff wrote:
>> If the nvme subsyetm reset causes the loss of communication to the nvme
>> adapter then EEH could potnetially recover the adapter. The detection of
>> comminication loss to the adapter only happens when the nvme driver
>> attempts to read an MMIO register.
>>
>> The nvme subsystem reset command writes 0x4E564D65 to NSSR register and
>> schedule adapter reset.In the case nvme subsystem reset caused the loss
>> of communication to the nvme adapter then either IO timeout event or
>> adapter reset handler could detect it. If IO timeout even could detect
>> loss of communication then EEH handler is able to recover the
>> communication to the adapter. This change was implemented in 651438bb0af5
>> (nvme-pci: Fix EEH failure on ppc). However if the adapter communication
>> loss is detected in nvme reset work handler then EEH is unable to
>> successfully finish the adapter recovery.
>>
>> This patch ensures that,
>> - nvme driver reset handler would observer pci channel was offline after
>>    a failed MMIO read and avoids marking the controller state to DEAD and
>>    thus gives a fair chance to EEH handler to recover the nvme adapter.
>>
>> - if nvme controller is already in RESETTNG state and pci channel frozen
>>    error is detected then  nvme driver pci-error-handler code sends the
>>    correct error code (PCI_ERS_RESULT_NEED_RESET) back to the EEH handler
>>    so that EEH handler could proceed with the pci slot reset.
>>
>> Signed-off-by: Nilay Shroff<nilay@linux.ibm.com>
>>
>> [  131.415601] EEH: Recovering PHB#40-PE#10000
>> [  131.415619] EEH: PE location: N/A, PHB location: N/A
>> [  131.415623] EEH: Frozen PHB#40-PE#10000 detected
>> [  131.415627] EEH: Call Trace:
>> [  131.415629] EEH: [c000000000051078] __eeh_send_failure_event+0x7c/0x15c
>> [  131.415782] EEH: [c000000000049bdc] eeh_dev_check_failure.part.0+0x27c/0x6b0
>> [  131.415789] EEH: [c000000000cb665c] nvme_pci_reg_read32+0x78/0x9c
>> [  131.415802] EEH: [c000000000ca07f8] nvme_wait_ready+0xa8/0x18c
>> [  131.415814] EEH: [c000000000cb7070] nvme_dev_disable+0x368/0x40c
>> [  131.415823] EEH: [c000000000cb9970] nvme_reset_work+0x198/0x348
>> [  131.415830] EEH: [c00000000017b76c] process_one_work+0x1f0/0x4f4
>> [  131.415841] EEH: [c00000000017be2c] worker_thread+0x3bc/0x590
>> [  131.415846] EEH: [c00000000018a46c] kthread+0x138/0x140
>> [  131.415854] EEH: [c00000000000dd58] start_kernel_thread+0x14/0x18
>> [  131.415864] EEH: This PCI device has failed 1 times in the last hour and will be permanently disabled after 5 failures.
>> [  131.415874] EEH: Notify device drivers to shutdown
>> [  131.415882] EEH: Beginning: 'error_detected(IO frozen)'
>> [  131.415888] PCI 0040:01:00.0#10000: EEH: Invoking nvme->error_detected(IO frozen)
>> [  131.415891] nvme nvme1: frozen state error detected, reset controller
>> [  131.515358] nvme 0040:01:00.0: enabling device (0000 -> 0002)
>> [  131.515778] nvme nvme1: Disabling device after reset failure: -19
>> [  131.555336] PCI 0040:01:00.0#10000: EEH: nvme driver reports: 'disconnect'
>> [  131.555343] EEH: Finished:'error_detected(IO frozen)' with aggregate recovery state:'disconnect'
>> [  131.555371] EEH: Unable to recover from failure from PHB#40-PE#10000.
>> [  131.555371] Please try reseating or replacing it
>> [  131.556296] EEH: of node=0040:01:00.0
>> [  131.556351] EEH: PCI device/vendor: 00251e0f
>> [  131.556421] EEH: PCI cmd/status register: 00100142
>> [  131.556428] EEH: PCI-E capabilities and status follow:
>> [  131.556678] EEH: PCI-E 00: 0002b010 10008fe3 00002910 00436044
>> [  131.556859] EEH: PCI-E 10: 10440000 00000000 00000000 00000000
>> [  131.556869] EEH: PCI-E 20: 00000000
>> [  131.556875] EEH: PCI-E AER capability register set follows:
>> [  131.557115] EEH: PCI-E AER 00: 14820001 00000000 00400000 00462030
>> [  131.557294] EEH: PCI-E AER 10: 00000000 0000e000 000002a0 00000000
>> [  131.557469] EEH: PCI-E AER 20: 00000000 00000000 00000000 00000000
>> [  131.557523] EEH: PCI-E AER 30: 00000000 00000000
>> [  131.558807] EEH: Beginning: 'error_detected(permanent failure)'
>> [  131.558815] PCI 0040:01:00.0#10000: EEH: Invoking nvme->error_detected(permanent failure)
>> [  131.558818] nvme nvme1: failure state error detected, request disconnect
>> [  131.558839] PCI 0040:01:00.0#10000: EEH: nvme driver reports: 'disconnect'
>> ---
>>   drivers/nvme/host/pci.c | 16 +++++++++++++---
>>   1 file changed, 13 insertions(+), 3 deletions(-)
>>
>> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
>> index c1d6357ec98a..a6ba46e727ba 100644
>> --- a/drivers/nvme/host/pci.c
>> +++ b/drivers/nvme/host/pci.c
>> @@ -2776,6 +2776,14 @@ static void nvme_reset_work(struct work_struct *work)
>>    out_unlock:
>>       mutex_unlock(&dev->shutdown_lock);
>>    out:
>> +    /*
>> +     * If PCI recovery is ongoing then let it finish first
>> +     */
>> +    if (pci_channel_offline(to_pci_dev(dev->dev))) {
>> +        dev_warn(dev->ctrl.device, "PCI recovery is ongoing so let it finish\n");
>> +        return;
>> +    }
>> +
>>       /*
>>        * Set state to deleting now to avoid blocking nvme_wait_reset(), which
>>        * may be holding this pci_dev's device lock.
>> @@ -3295,9 +3303,11 @@ static pci_ers_result_t nvme_error_detected(struct pci_dev *pdev,
>>       case pci_channel_io_frozen:
>>           dev_warn(dev->ctrl.device,
>>               "frozen state error detected, reset controller\n");
>> -        if (!nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_RESETTING)) {
>> -            nvme_dev_disable(dev, true);
>> -            return PCI_ERS_RESULT_DISCONNECT;
>> +        if (nvme_ctrl_state(&dev->ctrl) != NVME_CTRL_RESETTING) {
>> +            if (!nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_RESETTING)) {
>> +                nvme_dev_disable(dev, true);
>> +                return PCI_ERS_RESULT_DISCONNECT;
>> +            }
>>           }
>>           nvme_dev_disable(dev, false);
>>           return PCI_ERS_RESULT_NEED_RESET;