[PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[ojab]

MPI2 controllers sometimes got lost (i. e. disappears from
/sys/bus/pci/devices) if ASMP is enabled.

Signed-off-by: Slava Kardakov <ojab@ojab.ru>
Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=60644
---
V2: use name in Signed-off-by

Not sure if it's a complete fix, but at least I can't reproduce the issue
locally with it applied.

Also it's my first patch, so I've surely screwed up some formatting etc.

 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index b5c966e..203651a 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -51,6 +51,7 @@
 #include <linux/workqueue.h>
 #include <linux/delay.h>
 #include <linux/pci.h>
+#include <linux/pci-aspm.h>
 #include <linux/interrupt.h>
 #include <linux/aer.h>
 #include <linux/raid_class.h>
@@ -8734,6 +8735,8 @@ _scsih_probe(struct pci_dev *pdev, const struct pci_device_id *id)

 	switch (hba_mpi_version) {
 	case MPI2_VERSION:
+		pci_disable_link_state(pdev, PCIE_LINK_STATE_L0S |
+			PCIE_LINK_STATE_L1 | PCIE_LINK_STATE_CLKPM);
 		/* Use mpt2sas driver host template for SAS 2.0 HBA's */
 		shost = scsi_host_alloc(&mpt2sas_driver_template,
 		  sizeof(struct MPT3SAS_ADAPTER));
--
2.10.0

Re: [PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[Martin K. Petersen]

>>>>> "ojab" == ojab  <ojab@ojab.ru> writes:

ojab> MPI2 controllers sometimes got lost (i. e. disappears from
ojab> /sys/bus/pci/devices) if ASMP is enabled.

Broadcom folks: Please comment!

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[Sreekanth Reddy]

On Fri, Jan 6, 2017 at 7:24 AM, Martin K. Petersen
<martin.petersen@oracle.com> wrote:
>>>>>> "ojab" == ojab  <ojab@ojab.ru> writes:
>
> ojab> MPI2 controllers sometimes got lost (i. e. disappears from
> ojab> /sys/bus/pci/devices) if ASMP is enabled.
>
> Broadcom folks: Please comment!

Matin, We need some time to review this patch. I will provide my
review comments by end of next week.

Thanks,
Sreekanth
>
> --
> Martin K. Petersen      Oracle Linux Engineering
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[ojab]

On 2017/01/06 15:48, Sreekanth Reddy wrote:
> On Fri, Jan 6, 2017 at 7:24 AM, Martin K. Petersen
> <martin.petersen@oracle.com> wrote:
>
> Matin, We need some time to review this patch. I will provide my
> review comments by end of next week.
>
ping?

//wbr ojab
> Thanks,
> Sreekanth

Re: [PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[Sreekanth Reddy]

Ojab,

I am checking internally with our FW team and Architect to known
whether they aware/observed any issues similar to this (since this
issue is nothing to do with driver).

Meanwhile is it possible to provide us the PCI config space and FW
logs when issue occurs?

Thanks,
Sreekanth

On Mon, Jan 16, 2017 at 5:37 PM, ojab <ojab@ojab.ru> wrote:
> On 2017/01/06 15:48, Sreekanth Reddy wrote:
>>
>> On Fri, Jan 6, 2017 at 7:24 AM, Martin K. Petersen
>> <martin.petersen@oracle.com> wrote:
>>
>> Matin, We need some time to review this patch. I will provide my
>> review comments by end of next week.
>>
> ping?
>
> //wbr ojab
>>
>> Thanks,
>> Sreekanth
>
>

Re: [PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[ojab]

On 2017/01/16 12:36, Sreekanth Reddy wrote:
> Ojab,
>
> I am checking internally with our FW team and Architect to known
> whether they aware/observed any issues similar to this (since this
> issue is nothing to do with driver).
>
> Meanwhile is it possible to provide us the PCI config space and FW
> logs when issue occurs?

AFAIU FW log is enabled when `logging_level=0x3f8` is passed to the 
module, right?
Where can I find info about PCI config space?

//wbr ojab
>
> Thanks,
> Sreekanth
>
> On Mon, Jan 16, 2017 at 5:37 PM, ojab <ojab@ojab.ru> wrote:
>> On 2017/01/06 15:48, Sreekanth Reddy wrote:
>>>
>>> On Fri, Jan 6, 2017 at 7:24 AM, Martin K. Petersen
>>> <martin.petersen@oracle.com> wrote:
>>>
>>> Matin, We need some time to review this patch. I will provide my
>>> review comments by end of next week.
>>>
>> ping?
>>
>> //wbr ojab
>>>
>>> Thanks,
>>> Sreekanth
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[Sreekanth Reddy]

On Mon, Jan 16, 2017 at 6:15 PM, ojab <ojab@ojab.ru> wrote:
> On 2017/01/16 12:36, Sreekanth Reddy wrote:
>>
>> Ojab,
>>
>> I am checking internally with our FW team and Architect to known
>> whether they aware/observed any issues similar to this (since this
>> issue is nothing to do with driver).
>>
>> Meanwhile is it possible to provide us the PCI config space and FW
>> logs when issue occurs?
>
>
> AFAIU FW log is enabled when `logging_level=0x3f8` is passed to the module,
> right?

This is a driver logging_level. Firmware logs can be collected using UART logs.

> Where can I find info about PCI config space?

You can get PCI config space info from lspci command.

>
> //wbr ojab
>>
>>
>> Thanks,
>> Sreekanth
>>
>> On Mon, Jan 16, 2017 at 5:37 PM, ojab <ojab@ojab.ru> wrote:
>>>
>>> On 2017/01/06 15:48, Sreekanth Reddy wrote:
>>>>
>>>>
>>>> On Fri, Jan 6, 2017 at 7:24 AM, Martin K. Petersen
>>>> <martin.petersen@oracle.com> wrote:
>>>>
>>>> Matin, We need some time to review this patch. I will provide my
>>>> review comments by end of next week.
>>>>
>>> ping?
>>>
>>> //wbr ojab
>>>>
>>>>
>>>> Thanks,
>>>> Sreekanth
>>>
>>>
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

Re: [PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[ojab]

On 2017/01/16 13:31, Sreekanth Reddy wrote:
> On Mon, Jan 16, 2017 at 6:15 PM, ojab <ojab@ojab.ru> wrote:
>> On 2017/01/16 12:36, Sreekanth Reddy wrote:
>>>
>>> Ojab,
>>>
>>> I am checking internally with our FW team and Architect to known
>>> whether they aware/observed any issues similar to this (since this
>>> issue is nothing to do with driver).
>>>
>>> Meanwhile is it possible to provide us the PCI config space and FW
>>> logs when issue occurs?
>>
>>
>> AFAIU FW log is enabled when `logging_level=0x3f8` is passed to the module,
>> right?
>
> This is a driver logging_level. Firmware logs can be collected using UART logs.

According to the user guide (from 
https://www.broadcom.com/products/storage/host-bus-adapters/sas-9217-8i#documentation) 
my 9217-8i requires 1.8V UART which I don't have at hand, I'll try to do 
something on the weekend.
Is anything needed except connecting UART to gather required info?

Anyway, right now various MPI2 HBAs doesn't work with ASPM enabled, so 
can this patch be applied in the meantime while we're debugging the 
issue further? Even if this issue can be properly fixed by FW update, 
this workaround will be needed for HBAs with affected FW versions.

//wbr ojab


>
>> Where can I find info about PCI config space?
>
> You can get PCI config space info from lspci command.
>
>>
>> //wbr ojab
>>>
>>>
>>> Thanks,
>>> Sreekanth
>>>
>>> On Mon, Jan 16, 2017 at 5:37 PM, ojab <ojab@ojab.ru> wrote:
>>>>
>>>> On 2017/01/06 15:48, Sreekanth Reddy wrote:
>>>>>
>>>>>
>>>>> On Fri, Jan 6, 2017 at 7:24 AM, Martin K. Petersen
>>>>> <martin.petersen@oracle.com> wrote:
>>>>>
>>>>> Matin, We need some time to review this patch. I will provide my
>>>>> review comments by end of next week.
>>>>>
>>>> ping?
>>>>
>>>> //wbr ojab
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Sreekanth
>>>>
>>>>
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[ojab]

On 2017/01/16 14:03, ojab wrote:
> On 2017/01/16 13:31, Sreekanth Reddy wrote:
>> On Mon, Jan 16, 2017 at 6:15 PM, ojab <ojab@ojab.ru> wrote:
>>> On 2017/01/16 12:36, Sreekanth Reddy wrote:
>>>>
>>>> Ojab,
>>>>
>>>> I am checking internally with our FW team and Architect to known
>>>> whether they aware/observed any issues similar to this (since this
>>>> issue is nothing to do with driver).
>>>>
>>>> Meanwhile is it possible to provide us the PCI config space and FW
>>>> logs when issue occurs?
>>>
>>>
>>> AFAIU FW log is enabled when `logging_level=0x3f8` is passed to the
>>> module,
>>> right?
>>
>> This is a driver logging_level. Firmware logs can be collected using
>> UART logs.
>
> According to the user guide (from
> https://www.broadcom.com/products/storage/host-bus-adapters/sas-9217-8i#documentation)
> my 9217-8i requires 1.8V UART which I don't have at hand, I'll try to do
> something on the weekend.
> Is anything needed except connecting UART to gather required info?
…actually I have compatible USB UART here. What should be done to get FW 
log? I've tried to reboot with UART connected (9600N1) but got no output.

//wbr ojab


>
> Anyway, right now various MPI2 HBAs doesn't work with ASPM enabled, so
> can this patch be applied in the meantime while we're debugging the
> issue further? Even if this issue can be properly fixed by FW update,
> this workaround will be needed for HBAs with affected FW versions.
>
> //wbr ojab
>
>
>>
>>> Where can I find info about PCI config space?
>>
>> You can get PCI config space info from lspci command.
>>
>>>
>>> //wbr ojab
>>>>
>>>>
>>>> Thanks,
>>>> Sreekanth
>>>>
>>>> On Mon, Jan 16, 2017 at 5:37 PM, ojab <ojab@ojab.ru> wrote:
>>>>>
>>>>> On 2017/01/06 15:48, Sreekanth Reddy wrote:
>>>>>>
>>>>>>
>>>>>> On Fri, Jan 6, 2017 at 7:24 AM, Martin K. Petersen
>>>>>> <martin.petersen@oracle.com> wrote:
>>>>>>
>>>>>> Matin, We need some time to review this patch. I will provide my
>>>>>> review comments by end of next week.
>>>>>>
>>>>> ping?
>>>>>
>>>>> //wbr ojab
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Sreekanth
>>>>>
>>>>>
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe
>>>> linux-scsi" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[Sreekanth Reddy]

On Tue, Jan 17, 2017 at 4:13 AM, ojab <ojab@ojab.ru> wrote:
> On 2017/01/16 14:03, ojab wrote:
>>
>> On 2017/01/16 13:31, Sreekanth Reddy wrote:
>>>
>>> On Mon, Jan 16, 2017 at 6:15 PM, ojab <ojab@ojab.ru> wrote:
>>>>
>>>> On 2017/01/16 12:36, Sreekanth Reddy wrote:
>>>>>
>>>>>
>>>>> Ojab,
>>>>>
>>>>> I am checking internally with our FW team and Architect to known
>>>>> whether they aware/observed any issues similar to this (since this
>>>>> issue is nothing to do with driver).
>>>>>
>>>>> Meanwhile is it possible to provide us the PCI config space and FW
>>>>> logs when issue occurs?
>>>>
>>>>
>>>>
>>>> AFAIU FW log is enabled when `logging_level=0x3f8` is passed to the
>>>> module,
>>>> right?
>>>
>>>
>>> This is a driver logging_level. Firmware logs can be collected using
>>> UART logs.
>>
>>
>> According to the user guide (from
>>
>> https://www.broadcom.com/products/storage/host-bus-adapters/sas-9217-8i#documentation)
>> my 9217-8i requires 1.8V UART which I don't have at hand, I'll try to do
>> something on the weekend.
>> Is anything needed except connecting UART to gather required info?
>
> …actually I have compatible USB UART here. What should be done to get FW
> log? I've tried to reboot with UART connected (9600N1) but got no output.

You have to use, 115200 instead of 9600 baud rate.

Thanks,
Sreekanth

>
> //wbr ojab
>
>
>>
>> Anyway, right now various MPI2 HBAs doesn't work with ASPM enabled, so
>> can this patch be applied in the meantime while we're debugging the
>> issue further? Even if this issue can be properly fixed by FW update,
>> this workaround will be needed for HBAs with affected FW versions.
>>
>> //wbr ojab
>>
>>
>>>
>>>> Where can I find info about PCI config space?
>>>
>>>
>>> You can get PCI config space info from lspci command.
>>>
>>>>
>>>> //wbr ojab
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Sreekanth
>>>>>
>>>>> On Mon, Jan 16, 2017 at 5:37 PM, ojab <ojab@ojab.ru> wrote:
>>>>>>
>>>>>>
>>>>>> On 2017/01/06 15:48, Sreekanth Reddy wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jan 6, 2017 at 7:24 AM, Martin K. Petersen
>>>>>>> <martin.petersen@oracle.com> wrote:
>>>>>>>
>>>>>>> Matin, We need some time to review this patch. I will provide my
>>>>>>> review comments by end of next week.
>>>>>>>
>>>>>> ping?
>>>>>>
>>>>>> //wbr ojab
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Sreekanth
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> linux-scsi" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

Re: [PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[ojab]

On 2017/01/17 14:15, Sreekanth Reddy wrote:
> On Tue, Jan 17, 2017 at 4:13 AM, ojab <ojab@ojab.ru> wrote:
>> On 2017/01/16 14:03, ojab wrote:
>>>
>>> On 2017/01/16 13:31, Sreekanth Reddy wrote:
>>>>
>>>> On Mon, Jan 16, 2017 at 6:15 PM, ojab <ojab@ojab.ru> wrote:
>>>>>
>>>>> On 2017/01/16 12:36, Sreekanth Reddy wrote:
>>>>>>
>>>>>>
>>>>>> Ojab,
>>>>>>
>>>>>> I am checking internally with our FW team and Architect to known
>>>>>> whether they aware/observed any issues similar to this (since this
>>>>>> issue is nothing to do with driver).
>>>>>>
>>>>>> Meanwhile is it possible to provide us the PCI config space and FW
>>>>>> logs when issue occurs?
>>>>>
>>>>>
>>>>>
>>>>> AFAIU FW log is enabled when `logging_level=0x3f8` is passed to the
>>>>> module,
>>>>> right?
>>>>
>>>>
>>>> This is a driver logging_level. Firmware logs can be collected using
>>>> UART logs.
>>>
>>>
>>> According to the user guide (from
>>>
>>> https://www.broadcom.com/products/storage/host-bus-adapters/sas-9217-8i#documentation)
>>> my 9217-8i requires 1.8V UART which I don't have at hand, I'll try to do
>>> something on the weekend.
>>> Is anything needed except connecting UART to gather required info?
>>
>> …actually I have compatible USB UART here. What should be done to get FW
>> log? I've tried to reboot with UART connected (9600N1) but got no output.
>
> You have to use, 115200 instead of 9600 baud rate.
No output using 115200 8N1 either.
If I connect Bus Pirate to the pins and enable baud autodetect, I see 
something like
 >Calculated: 	117647 bps
 >Estimated:  	115200 bps
on reboot and firmware/bios upgrade, but there is no text at all.
Right now I have IT firmware installed, `sas2flash -list` shows
 >	Adapter Selected is a LSI SAS: SAS2308_2(D1)
 >
 >	Controller Number              : 0
 >	Controller                     : SAS2308_2(D1)
 >	PCI Address                    : 00:02:00:00
 >	SAS Address                    : 500605b-0-05a6-c320
 >	NVDATA Version (Default)       : 14.01.00.06
 >	NVDATA Version (Persistent)    : 14.01.00.06
 >	Firmware Product ID            : 0x2214 (IT)
 >	Firmware Version               : 20.00.07.00
 >	NVDATA Vendor                  : LSI
 >	NVDATA Product ID              : SAS9207-8i
 >	BIOS Version                   : 07.39.02.00
 >	UEFI BSD Version               : 07.27.01.01
 >	FCODE Version                  : N/A
 >	Board Name                     : SAS9207-8i
 >	Board Assembly                 : N/A
 >	Board Tracer Number            : N/A

//wbr ojab

>
> Thanks,
> Sreekanth
>
>>
>> //wbr ojab
>>
>>
>>>
>>> Anyway, right now various MPI2 HBAs doesn't work with ASPM enabled, so
>>> can this patch be applied in the meantime while we're debugging the
>>> issue further? Even if this issue can be properly fixed by FW update,
>>> this workaround will be needed for HBAs with affected FW versions.
>>>
>>> //wbr ojab
>>>
>>>
>>>>
>>>>> Where can I find info about PCI config space?
>>>>
>>>>
>>>> You can get PCI config space info from lspci command.
>>>>
>>>>>
>>>>> //wbr ojab
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Sreekanth
>>>>>>
>>>>>> On Mon, Jan 16, 2017 at 5:37 PM, ojab <ojab@ojab.ru> wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 2017/01/06 15:48, Sreekanth Reddy wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jan 6, 2017 at 7:24 AM, Martin K. Petersen
>>>>>>>> <martin.petersen@oracle.com> wrote:
>>>>>>>>
>>>>>>>> Matin, We need some time to review this patch. I will provide my
>>>>>>>> review comments by end of next week.
>>>>>>>>
>>>>>>> ping?
>>>>>>>
>>>>>>> //wbr ojab
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Sreekanth
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> linux-scsi" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[ojab]

On 2016/12/28 11:05, ojab wrote:
> MPI2 controllers sometimes got lost (i. e. disappears from
> /sys/bus/pci/devices) if ASMP is enabled.
>
> Signed-off-by: Slava Kardakov <ojab@ojab.ru>
> Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=60644

Can I haz Reviewed-by?

//wbr ojab

Re: [PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[ojab]

On 2017/01/23 15:59, ojab wrote:
> On 2016/12/28 11:05, ojab wrote:
>> MPI2 controllers sometimes got lost (i. e. disappears from
>> /sys/bus/pci/devices) if ASMP is enabled.
>>
>> Signed-off-by: Slava Kardakov <ojab@ojab.ru>
>> Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=60644
>
> Can I haz Reviewed-by?
>

ping?

//wbr ojab

Re: [PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[ojab]

On 2017/01/30 10:58, ojab wrote:
> On 2017/01/23 15:59, ojab wrote:
>> On 2016/12/28 11:05, ojab wrote:
>>> MPI2 controllers sometimes got lost (i. e. disappears from
>>> /sys/bus/pci/devices) if ASMP is enabled.
>>>
>>> Signed-off-by: Slava Kardakov <ojab@ojab.ru>
>>> Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=60644
>>
>> Can I haz Reviewed-by?
>>
>
> ping?
>

Humble ping.

//wbr ojab

Re: [PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[Sreekanth Reddy]

Hi Ojab,

Can I get output of 'lspci -vvv -s' for LSI SAS2 card?

On my Supermicro system, I have enabled ASPM in system BIOS but still
in lspci output I see this ASPM was disabled for my SAS2 HBA card
which has Phase19 firmware (i.e. 19.00.00.00) as shown below,

# lspci -vvv -s 83:00.0
83:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic
SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
    Subsystem: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2
    Physical Slot: 1-1
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR+ FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 32 bytes
    Interrupt: pin A routed to IRQ 40
    Region 0: I/O ports at f000 [size=256]
    Region 1: Memory at fbe40000 (64-bit, non-prefetchable) [size=64K]
    Region 3: Memory at fbe00000 (64-bit, non-prefetchable) [size=256K]
    Expansion ROM at fbd00000 [disabled] [size=1M]
    Capabilities: [50] Power Management version 3
        Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [68] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 4096 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
        DevCtl:    Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
            RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
        LnkCap:    Port #0, Speed 8GT/s, Width x8, ASPM L0s, Exit
Latency L0s <64ns, L1 <1us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 8GT/s, Width x8, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF
Not Supported
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-,
OBFF Disabled
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete+, EqualizationPhase1+
             EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest+
    Capabilities: [d0] Vital Product Data
pcilib: sysfs_read_vpd: read failed: Input/output error
        Not readable
    Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [c0] MSI-X: Enable+ Count=16 Masked-
        Vector table: BAR=1 offset=0000e000
        PBA: BAR=1 offset=0000f000
    Capabilities: [100 v2] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        AERCap:    First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
    Capabilities: [1e0 v1] #19
    Capabilities: [1c0 v1] Power Budgeting <?>
    Capabilities: [190 v1] #16
    Capabilities: [148 v1] Alternative Routing-ID Interpretation (ARI)
        ARICap:    MFVC- ACS-, Next Function: 0
        ARICtl:    MFVC- ACS-, Function Group: 0
    Kernel driver in use: mpt3sas
    Kernel modules: mpt3sas

So, Can you also try once with Phase19 firmware.

Thanks,
Sreekanth

On Mon, Feb 6, 2017 at 2:22 PM, ojab <ojab@ojab.ru> wrote:
> On 2017/01/30 10:58, ojab wrote:
>>
>> On 2017/01/23 15:59, ojab wrote:
>>>
>>> On 2016/12/28 11:05, ojab wrote:
>>>>
>>>> MPI2 controllers sometimes got lost (i. e. disappears from
>>>> /sys/bus/pci/devices) if ASMP is enabled.
>>>>
>>>> Signed-off-by: Slava Kardakov <ojab@ojab.ru>
>>>> Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=60644
>>>
>>>
>>> Can I haz Reviewed-by?
>>>
>>
>> ping?
>>
>
> Humble ping.
>
> //wbr ojab

Re: [PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[ojab]

[-- Attachment #1: Type: text/plain, Size: 778 bytes --]

On 2017/02/06 12:02, Sreekanth Reddy wrote:
> Hi Ojab,
>
> Can I get output of 'lspci -vvv -s' for LSI SAS2 card?
>
> On my Supermicro system, I have enabled ASPM in system BIOS but still
> in lspci output I see this ASPM was disabled for my SAS2 HBA card
> which has Phase19 firmware (i.e. 19.00.00.00) as shown below,
>
[snip]
>
> So, Can you also try once with Phase19 firmware.

How can I downgrade firmware? Right now I have FW 20.00.07.00 installed 
(`lspci` output can be found in the attached file), `sas2flash -fwall 
[P19_firmaware.bin]` tells me:
 >ERROR: Cannot downgrade NVDATA version 14.01.00.06
 >       to 11.00.110000.00.
 >
 >ERROR: Failed to get valid NVDATA image from File!

and `sas2flash -e 6` tells me
 >ERROR: Erase Flash Operation Failed!

//wbr ojab

[-- Attachment #2: lspci_nopatch_P20.txt --]
[-- Type: text/plain, Size: 3202 bytes --]

02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
	Subsystem: LSI Logic / Symbios Logic Device 3020
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 19
	Region 0: I/O ports at c000 [size=256]
	Region 1: Memory at fe7f0000 (64-bit, non-prefetchable) [size=64K]
	Region 3: Memory at fe780000 (64-bit, non-prefetchable) [size=256K]
	Expansion ROM at fe600000 [disabled] [size=1M]
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM L0s Enabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [d0] Vital Product Data
		Not readable
	Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [c0] MSI-X: Enable+ Count=16 Masked-
		Vector table: BAR=1 offset=0000e000
		PBA: BAR=1 offset=0000f000
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr+ BadTLP- BadDLLP+ Rollover- Timeout+ NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [1e0 v1] #19
	Capabilities: [1c0 v1] Power Budgeting <?>
	Capabilities: [190 v1] #16
	Capabilities: [148 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Kernel driver in use: mpt3sas
	Kernel modules: mpt3sas

Re: [PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[Sreekanth Reddy]

On Wed, Dec 28, 2016 at 4:35 PM, ojab <ojab@ojab.ru> wrote:
> MPI2 controllers sometimes got lost (i. e. disappears from
> /sys/bus/pci/devices) if ASMP is enabled.
>
> Signed-off-by: Slava Kardakov <ojab@ojab.ru>
> Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=60644

>From some of our system engineers team, I came to known that ASPM
needs to be disabled.
So this patch looks good. Please consider this patch as

Acked-by: Sreekanth Reddy <Sreekanth.Reddy@broadcom.com>

Thanks,
Sreekanth

> ---
> V2: use name in Signed-off-by
>
> Not sure if it's a complete fix, but at least I can't reproduce the issue
> locally with it applied.
>
> Also it's my first patch, so I've surely screwed up some formatting etc.
>
>  drivers/scsi/mpt3sas/mpt3sas_scsih.c | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
> index b5c966e..203651a 100644
> --- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
> +++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
> @@ -51,6 +51,7 @@
>  #include <linux/workqueue.h>
>  #include <linux/delay.h>
>  #include <linux/pci.h>
> +#include <linux/pci-aspm.h>
>  #include <linux/interrupt.h>
>  #include <linux/aer.h>
>  #include <linux/raid_class.h>
> @@ -8734,6 +8735,8 @@ _scsih_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>
>         switch (hba_mpi_version) {
>         case MPI2_VERSION:
> +               pci_disable_link_state(pdev, PCIE_LINK_STATE_L0S |
> +                       PCIE_LINK_STATE_L1 | PCIE_LINK_STATE_CLKPM);
>                 /* Use mpt2sas driver host template for SAS 2.0 HBA's */
>                 shost = scsi_host_alloc(&mpt2sas_driver_template,
>                   sizeof(struct MPT3SAS_ADAPTER));
> --
> 2.10.0
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[ojab]

On 2017/02/09 05:09, Sreekanth Reddy wrote:
> On Wed, Dec 28, 2016 at 4:35 PM, ojab <ojab@ojab.ru> wrote:
>> MPI2 controllers sometimes got lost (i. e. disappears from
>> /sys/bus/pci/devices) if ASMP is enabled.
>>
>> Signed-off-by: Slava Kardakov <ojab@ojab.ru>
>> Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=60644
>
> From some of our system engineers team, I came to known that ASPM
> needs to be disabled.
> So this patch looks good. Please consider this patch as
>
> Acked-by: Sreekanth Reddy <Sreekanth.Reddy@broadcom.com>

Yay, thanks!

I assume that this patch should also be CC: stable@, since it's clearly 
a rather critical bugfix.

Is it too late for 4.10.0?

//wbr ojab
>
> Thanks,
> Sreekanth

Re: [PATCH V2] mpt3sas: disable ASPM for MPI2 controllers

[Martin K. Petersen]

>>>>> "ojab" == ojab  <ojab@ojab.ru> writes:

>> Acked-by: Sreekanth Reddy <Sreekanth.Reddy@broadcom.com>

Applied to 4.10/scsi-fixes.

ojab> Is it too late for 4.10.0?

We'll see. Worst case I'll shuffle it over to the 4.11 tree.

-- 
Martin K. Petersen	Oracle Linux Engineering