aacraid: kernel: AAC: Host adapter dead -1 (bisected)

aacraid: kernel: AAC: Host adapter dead -1 (bisected)

[Arkadiusz Miskiewicz]

Hi.

There is a bug with handling of adaptec raid cards (in my case it is Adaptec 
3405) where kernel logs hundreds of "AAC: Host adapter dead -1" messages.

Bug was reported previously on lkml but there was no progres in solving it.

There is also bugzilla entry:
https://bugzilla.kernel.org/show_bug.cgi?id=151661

I've bisected that to commit bellow and indeed, reverting it from kernel 4.9.3 
makes messages go away.

Could anyone at microsemi look at this regression?

Thanks

commit 78cbccd3bd683c295a44af8050797dc4a41376ff
Author: Raghava Aditya Renukunta <RaghavaAditya.Renukunta@microsemi.com>
Date:   Mon Apr 25 23:32:37 2016 -0700

    aacraid: Fix for KDUMP driver hang

    When KDUMP is triggered the driver first talks to the firmware in INTX
    mode, but the adapter firmware is still in MSIX mode. Therefore the first
    driver command hangs since the driver is waiting for an INTX response and
    firmware gives a MSIX response. If when the OS is installed on a RAID
    drive created by the adapter KDUMP will hang since the driver does not
    receive a response in sync mode.

    Fixed by: Change the firmware to INTX mode if it is in MSIX mode before
    sending the first sync command.

    Cc: stable@vger.kernel.org
    Signed-off-by: Raghava Aditya Renukunta 
<RaghavaAditya.Renukunta@microsemi.com>
    Reviewed-by: Johannes Thumshirn <jthumshirn@suse.de>
    Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

my hardware:
02:0e.0 RAID bus controller [0104]: Adaptec AAC-RAID [9005:0285]
        Subsystem: Adaptec 3405 [9005:02bb]
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping+ SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- 
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 32 (250ns min, 250ns max), Cache Line Size: 4 bytes
        Interrupt: pin A routed to IRQ 18
        Region 0: Memory at fba00000 (64-bit, non-prefetchable) [size=2M]
        [virtual] Expansion ROM at fbc00000 [disabled] [size=256K]
        Capabilities: [c0] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA 
PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [d0] MSI: Enable- Count=1/2 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [e0] PCI-X non-bridge device
                Command: DPERE- ERO- RBC=512 OST=4
                Status: Dev=02:0e.0 64bit+ 133MHz+ SCD- USC- DC=bridge 
DMMRBC=1024 DMOST=4 DMCRS=16 RSCEM- 266MHz- 533MHz-
        Kernel driver in use: aacraid
        Kernel modules: aacraid

[    1.956009] Adaptec aacraid driver 1.2-1[41066]-ms
[    2.164584] AAC0: kernel 5.2-0[17342] Aug  4 2010
[    2.164633] AAC0: monitor 5.2-0[17342]
[    2.164676] AAC0: bios 5.2-0[17342]
[    2.164719] AAC0: serial 7C46114103A
[    2.164761] AAC0: Non-DASD support enabled.
[    2.164804] AAC0: 64bit support enabled.
[    2.164846] AAC0: 64 Bit DAC enabled
[    2.177929] scsi host6: aacraid


-- 
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )

RE: aacraid: kernel: AAC: Host adapter dead -1 (bisected)

[Dave Carroll]

> 
> 
> Hi.
> 
> There is a bug with handling of adaptec raid cards (in my case it is Adaptec
> 3405) where kernel logs hundreds of "AAC: Host adapter dead -1"
> messages.
> 
> Bug was reported previously on lkml but there was no progres in solving it.
> 
> There is also bugzilla entry:
> https://bugzilla.kernel.org/show_bug.cgi?id=151661
> 
> I've bisected that to commit bellow and indeed, reverting it from kernel
> 4.9.3 makes messages go away.
> 
> Could anyone at microsemi look at this regression?
> 
> Thanks

Hi Arkadiusz,

Thanks for your effort in determining the cause of the issue. It makes sense now that the patch should have been included in controller specific code, rather than common code.

I will prepare a patch for this, and if you are willing to test it, that would be great!

Thanks, -Dave

> 
> commit 78cbccd3bd683c295a44af8050797dc4a41376ff
> Author: Raghava Aditya Renukunta
> <RaghavaAditya.Renukunta@microsemi.com>
> Date:   Mon Apr 25 23:32:37 2016 -0700
> 
>     aacraid: Fix for KDUMP driver hang
> 
>     When KDUMP is triggered the driver first talks to the firmware in INTX
>     mode, but the adapter firmware is still in MSIX mode. Therefore the
> first
>     driver command hangs since the driver is waiting for an INTX response
> and
>     firmware gives a MSIX response. If when the OS is installed on a RAID
>     drive created by the adapter KDUMP will hang since the driver does not
>     receive a response in sync mode.
> 
>     Fixed by: Change the firmware to INTX mode if it is in MSIX mode before
>     sending the first sync command.
> 
>     Cc: stable@vger.kernel.org
>     Signed-off-by: Raghava Aditya Renukunta
> <RaghavaAditya.Renukunta@microsemi.com>
>     Reviewed-by: Johannes Thumshirn <jthumshirn@suse.de>
>     Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
> 
> my hardware:
> 02:0e.0 RAID bus controller [0104]: Adaptec AAC-RAID [9005:0285]
>         Subsystem: Adaptec 3405 [9005:02bb]
>         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr-
> Stepping+ SERR+ FastB2B- DisINTx-
>         Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 32 (250ns min, 250ns max), Cache Line Size: 4 bytes
>         Interrupt: pin A routed to IRQ 18
>         Region 0: Memory at fba00000 (64-bit, non-prefetchable) [size=2M]
>         [virtual] Expansion ROM at fbc00000 [disabled] [size=256K]
>         Capabilities: [c0] Power Management version 2
>                 Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
>         Capabilities: [d0] MSI: Enable- Count=1/2 Maskable- 64bit+
>                 Address: 0000000000000000  Data: 0000
>         Capabilities: [e0] PCI-X non-bridge device
>                 Command: DPERE- ERO- RBC=512 OST=4
>                 Status: Dev=02:0e.0 64bit+ 133MHz+ SCD- USC- DC=bridge
> DMMRBC=1024 DMOST=4 DMCRS=16 RSCEM- 266MHz- 533MHz-
>         Kernel driver in use: aacraid
>         Kernel modules: aacraid
> 
> [    1.956009] Adaptec aacraid driver 1.2-1[41066]-ms
> [    2.164584] AAC0: kernel 5.2-0[17342] Aug  4 2010
> [    2.164633] AAC0: monitor 5.2-0[17342]
> [    2.164676] AAC0: bios 5.2-0[17342]
> [    2.164719] AAC0: serial 7C46114103A
> [    2.164761] AAC0: Non-DASD support enabled.
> [    2.164804] AAC0: 64bit support enabled.
> [    2.164846] AAC0: 64 Bit DAC enabled
> [    2.177929] scsi host6: aacraid
> 
> 
> --
> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )

Re: aacraid: kernel: AAC: Host adapter dead -1 (bisected)

[Arkadiusz Miskiewicz]

On Tuesday 17 of January 2017, Dave Carroll wrote:
> > Hi.
> > 
> > There is a bug with handling of adaptec raid cards (in my case it is
> > Adaptec 3405) where kernel logs hundreds of "AAC: Host adapter dead -1"
> > messages.
> > 
> > Bug was reported previously on lkml but there was no progres in solving
> > it.
> > 
> > There is also bugzilla entry:
> > https://bugzilla.kernel.org/show_bug.cgi?id=151661
> > 
> > I've bisected that to commit bellow and indeed, reverting it from kernel
> > 4.9.3 makes messages go away.
> > 
> > Could anyone at microsemi look at this regression?
> > 
> > Thanks
> 
> Hi Arkadiusz,
> 
> Thanks for your effort in determining the cause of the issue. It makes
> sense now that the patch should have been included in controller specific
> code, rather than common code.
> 
> I will prepare a patch for this, and if you are willing to test it, that
> would be great!

Great!

I have dedicated machine for testing this, so yes - I'll test.

-- 
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )

Re: aacraid: kernel: AAC: Host adapter dead -1 (bisected)

[Andrey Melnikov]

Cc: linux-scsi@vger.kernel.org

2017-02-10 13:24 GMT+03:00 Greg Kroah-Hartman <gregkh@linuxfoundation.org>:
> On Fri, Feb 10, 2017 at 02:25:26AM +0300, Andrey Jr. Melnikov wrote:
>> In article <201701151205.37563.a.miskiewicz@gmail.com> you wrote:
>> > Newsgroups: gmane.linux.kernel
>>
>>
>> > Hi.
>>
>> > There is a bug with handling of adaptec raid cards (in my case it is Adaptec
>> > 3405) where kernel logs hundreds of "AAC: Host adapter dead -1" messages.
>>
>> > Bug was reported previously on lkml but there was no progres in solving it.
>>
>> > There is also bugzilla entry:
>> > https://bugzilla.kernel.org/show_bug.cgi?id=151661
>>
>> > I've bisected that to commit bellow and indeed, reverting it from kernel 4.9.3
>> > makes messages go away.
>>
>>
>> Don't try to switch Adaptec 3405/3805 RAID cards to MSI-X interrupt mode.
>> Fix https://bugzilla.kernel.org/show_bug.cgi?id=151661
>>
>> Signed-off-by: Andrey Jr. Melnikov <temnota.am@gmail.com>
>>
>> ---
>>
>> diff --git a/drivers/scsi/aacraid/aacraid.h b/drivers/scsi/aacraid/aacraid.h
>> index 969c312de1be..2ad8403dea40 100644
>> --- a/drivers/scsi/aacraid/aacraid.h
>> +++ b/drivers/scsi/aacraid/aacraid.h
>
> <snip>
>
> Why are you sending this to me and not the scsi developers who can
> actually do something with this patch?

Bug in bugzilla open half year ago, microsemi maintainer slowly read
his fine docs about his hardware, broken driver fills our log with
useless messages every 10 seconds.
So, make decision - apply this patch to stable 4.9.x/4.4.x tree or
revert commit 78cbccd3bd683c295a44af8050797dc4a41376ff from it.

Re: aacraid: kernel: AAC: Host adapter dead -1 (bisected)

[Greg Kroah-Hartman]

On Fri, Feb 10, 2017 at 01:45:06PM +0300, Andrey Melnikov wrote:
> Cc: linux-scsi@vger.kernel.org
> 
> 2017-02-10 13:24 GMT+03:00 Greg Kroah-Hartman <gregkh@linuxfoundation.org>:
> > On Fri, Feb 10, 2017 at 02:25:26AM +0300, Andrey Jr. Melnikov wrote:
> >> In article <201701151205.37563.a.miskiewicz@gmail.com> you wrote:
> >> > Newsgroups: gmane.linux.kernel
> >>
> >>
> >> > Hi.
> >>
> >> > There is a bug with handling of adaptec raid cards (in my case it is Adaptec
> >> > 3405) where kernel logs hundreds of "AAC: Host adapter dead -1" messages.
> >>
> >> > Bug was reported previously on lkml but there was no progres in solving it.
> >>
> >> > There is also bugzilla entry:
> >> > https://bugzilla.kernel.org/show_bug.cgi?id=151661
> >>
> >> > I've bisected that to commit bellow and indeed, reverting it from kernel 4.9.3
> >> > makes messages go away.
> >>
> >>
> >> Don't try to switch Adaptec 3405/3805 RAID cards to MSI-X interrupt mode.
> >> Fix https://bugzilla.kernel.org/show_bug.cgi?id=151661
> >>
> >> Signed-off-by: Andrey Jr. Melnikov <temnota.am@gmail.com>
> >>
> >> ---
> >>
> >> diff --git a/drivers/scsi/aacraid/aacraid.h b/drivers/scsi/aacraid/aacraid.h
> >> index 969c312de1be..2ad8403dea40 100644
> >> --- a/drivers/scsi/aacraid/aacraid.h
> >> +++ b/drivers/scsi/aacraid/aacraid.h
> >
> > <snip>
> >
> > Why are you sending this to me and not the scsi developers who can
> > actually do something with this patch?
> 
> Bug in bugzilla open half year ago, microsemi maintainer slowly read
> his fine docs about his hardware, broken driver fills our log with
> useless messages every 10 seconds.
> So, make decision - apply this patch to stable 4.9.x/4.4.x tree or
> revert commit 78cbccd3bd683c295a44af8050797dc4a41376ff from it.

I don't understand, that's not how the stable kernels work, please read
Documentation/stable_kernel_rules.txt for how the process works.  Please
get a patch accepted into Linus's tree and then we will be glad to apply
it to the stable kernel trees.

thanks,

greg k-h